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Section  1.  Systems  Specification  (A  Spec) 

1-1.  Review  MIL-STD-483  and  MIL-STD-490,  particularly  Appendix  III  of 
MIL-STD-483  and  MIL-STD-490  of  Appendix  I. 

1-2.  Review  Section  2  (Pages  11-15)  of  ESD-TR-77-326,  Software  Acquisition 
Management  Guidebook:  Validation  and  Certification. 

1-3.  Review  and  modify  as  required  Figure  II-1-2,  Model  Paragraph  3  -  3 -  & , 
System  Specification. 

1-4.  Review  Figure  11-1-1,  System  Specification  Checklist  for  Computer 

Resources. 

1-5.  Review  and  modify  as  required  Figure  11-1-3,  Model  System  Specification 
Section  Four. 
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Figure  II-l-l.  System  Specification  Checklist  for  Computer  Resources 
NOTE:  Items  succeeded  by  an  (M)  are  mandatory. 

KM).  Does  tne  system  specification  contain  tDe  mandatory  inputs  listed 
in  ESD/ALr.y's  Model  Para  3.3-8? 

uoes  tne  system  specification  address  any  sizing  and  timing 
requirements?  (Usually  found  in  Paragraph  3-2.1  or  Paragraph  3-7,  in 
subparagrapns  pertaining  to  computer  equipment  and  peripherals.) 

3.  Have  software  performance  requirements  been  functionally  and/or 
quantitatively  defined  (i.e.,  to  allow  for  testing  compliance)? 

4(M).  Does  tne  system  specification  require  use  of  an  Air  Force  approved 
Hign  Order  Language?  ( AFR  300-10  including  AFSC  Sup  1;  also  listed  in  model 
paragraph  3-3-8  for  System  Spec.) 

5.  Are  test  methods  in  Section  4  well-defined  (see  Figure  II-1-3-)? 
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Figure  II-1-2.  Model  Paragraph  3-3*8,  System  Specification 

This  general  sample  contains  minimum  essential  requirements  and  is 
intended  to  serve  as  guidance  for  composing  a  System  "A" 

Specification  Section  3-3-8.  Requirements  stemming  from  DOD  or  AP 
regulations  are  indicated  by  an  M  in  the  left  margin.  These  are 
mandatory  inputs  to  the  System  Specif ication.  (This  is  only  for 
information  purposes  -  the  M  does  not  go  in  the  final  specification.) 

3-3-8  Computer  Programming.  Computer  programs  and  computer  data  bases  shall 
be  considered  as  software.  Software  shall  be  categorized  as  support  or 
applications  software. 

3- 3-8.1  General  Requirements.  Software  shall  meet  the  following  design, 
language,  and  coding  requirements : 

3. 3-  8. 1.1  Design  Requirements 

3. 3- 8. 1.1.1  Computer  Program  Structure.  The  computer  program  structi  shall 
consist  of  Computer  Program  Configuration  Item(s),  Computer  Program 
Component (s ) ,  and  Module(s). 

a.  Computer  Program  Configuration  Item  (CPCI).  A  CPCI  is  the  ac  a 
computer  program  end  item  in  the  form  of  computer  instructions  stored 
machine-readable  media.  A  CPCI  shall  consist  of  one  or  more  computer  program 
components. 

b.  Computer  Program  Component  (CPC).  A  CPC  is  a  functionally,  logically 
distinct  part  of  a  CPCI.  A  CPC  is  identified  for  purposes  of  convenience  in 
specifying  and  developing  a  CPCI  as  an  assembly  of  subordinate  elements.  A 
CPC  consists  of  a  logical  composition  of  one  or  more  subordinate  or 
interfacing  modules. 

c.  Module.  A  module  performs  a  complete  logical  process  by  execution  of 
a  set  of  instructions  which  have  clearly  defined  inputs,  processing  logic  and 
outputs.  A  module  is  the  smallest  set  of  executable  statements  able  to  be 
assembled  or  compiled.  Each  module  shall  conform  to  the  following  conventions: 

(1)  A  module  shall  consist  of  a  set  of  instructions  in  a  form 
consistent  with  the  appropriate  language,  operating  system,  and  computer. 

(2)  A  module  shall  not  exceed  100  lines  of  executable  source  code. 

This  limitation  excludes  comments  and  data  definitions. 

(3)  A  module  shall  have  only  one  entry  statement  and  one  exit 
statement . 

3- 3-8. 1.1. 2  Top  Down  Design  (TDD).  Software  shall  be  designed  in  a  top  down 
manner.  The  processing  activities  of  the  system  shall  be  identified  and 
organized  beginning  with  higher  levels  of  organization,  i.e.,  top  levels. 

These  higher  levels  shall  then  be  expanded  and  broken  out  to  include  a  more 
detailed  definition  of  the  processing  activities  by  identification  of 
subordinate  levels.  The  lowest  level  of  processing  shall  correspond  to  the 
modu le . 
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i.I.  5  lop  Down  Implementation  i TL> I J .  The  project  software  snail  be 
implemented  in  a  top  down  manner  as  defined  nerein.  Conceptually,  top  down 
implementation  proceeds  from  a  single  starting  point  while  conventional 
implementation  proceeds  from  as  many  starting  points  as  programs  in  the 
design.  Tne  single  starting  point  does  not  imply  that  the  implementation  must 
proceed  down  tne  nierarcny  in  parallel.  Some  branches  intentionally  will  pe 
developed  earlier  tnan  otner  branches.  For  example,  user  or  other  external 
interfaces  might  be  implemented  before  some  of  tne  other  partitions  to  permit 
early  demonstration  of  software  subsystem  capabilities,  partial  software 
system  evaluation,  training,  or  even  incremental  software  system  acceptance. 
Tne  project  software  snail  be  implemented  in  a  series  of  RELEASES  which  shall 
provide  for  successive  system  capabilities. 

(M; <o. 6. 1 . H  Programming  Languages.  Software  for  this  system  shall  be 
restricted  to  JuViAi.  Mi  as  per  MIL-STD-lSdyB,  except  for  automatic  test 
equipment,  wnere  AiuAS  as  per  IEEE  3Tb  7io-ld6c  is  required.  If  compelling 
justification  exists,  the  following  languages  are  allowed: 

a.  FORTRAN  as  per  ANSI  STL  Xj. 9-1973  (FORTRAN  77)  (with  or  without  tne 
addition  of  MIL-STL-1753) • 


o.  CObUu  as  per  FIPS  Plib  <11-1. 

j.j.O.l.j.  Structured  Coding  Requirements.  Computer  programs  coded  for  tne 
system  snail  employ  only  the  control  constructs  listed  below.  These 
constructs  shall  oe  built  using  logically  equivalent  language  simulations. 
Instructions  in  tne  language  used  shall  follow  the  graphic  representations  in 
Figure  1. 


a.  SEQUENCE.  Sequence  of  two  or  more  operations. 

o.  iF-THEN-ELSE.  Conditional  branch  to  one  of  two  mutually  exclusive 
operations  ana  continue. 

c.  DO-WHILb.  Operation  repeated  while  a  condition  is  true.  Test  is 
before  operation. 

d.  DO-tJNTiL.  Operation  repeated  until  a  condition  becomes  true.  Test  is 
after  operation. 

e.  CAbE.  Select  one  of  many  possible  cases. 

j.j.d.o  Operating  System  (OS)  Requirements.  The  OS  shall  conform  to  the 
following  requirements: 

a.  Tne  OS  shall  be  a  vendor-supplied,  off-the-snelf  package. 

d.  u3  augmentations  shall  be  allowed  but  shall  be  limited  to  new 
software.  No  augmentations  shall  be  permitted  to  be  embedded  witnin  the 
vendor  supplied  OS  software;  a  separate  interface  shall  be  provided. 

c.  No  OS  interface  or  augmentation  software  shall  compromise  the 
capability  of  the  OS  vendor  to  provide  maintenance  over  the  life  cycle  of  the 
systems. 
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d.  No  instructions  shall  be  executed  that  will  cause  the  computer  to  halt 
processing  pending  an  external  event,  except  by  the  OS.  An  exception  to  this 
restriction  snail  be  permitted  for  augmentations  to  the  OS  where  the 
augmentation  is  designed  as  an  extension  of  the  processing  control  of  the  OS. 
The  exception  is  subject  to  review  and  approval  by  the  Government. 

(M) 3* 3.8. 3  Firmware  Requirements.  Computer  programs  and  data  loaded  in  a 
class  of  memory  that  cannot  be  dynamically  modified  by  the  computer  during 
processing  shall  be  considered  firmware.  Requirements  on  firmware  shall  oe 
the  same  as  those  on  software.  Use  of  firmware  shall  be  subject  to  approval 
by  the  Government. 

3.3*8. 9  Software  Utility  Services.  This  support  software  shall  provide  the 
following  minimum  capabilities: 

a.  Compilation. 

b.  Assembly  wnich  produces  relocatable  object  code. 

c.  Linking  type  loader. 

d.  Generation,  maintenance,  and  initialization  of  storage  media  for 
programs  and  data. 

e.  Diagnostics  to  support  fault  isolation. 

f.  Editing  and  debugging  tools. 

3. 3.8.5  Message  Generation.  The  generation  of  error/diagnostic  messages 
shall  make  a  distinction  between  (1)  the  requirements  for  on-line  messages  to 
facilitate  real-time  fault  isolation  required  to  maintain  the  system  in 
operational  status  and  (2)  the  logging  of  fault  messages  onto  system  files 
for  the  category  of  faults  which  require  isolation  and  correction  but  can  be 
addressed  off-line  and  do  not  degrade  the  system  performance.  The  required 
processing  time  to  identify  and  generate  an  error/diagnostic  message  either 
for  on-line  or  off-line  isolation  and  correction  shall  not  degrade  the 
operational  requirements  of  the  system. 

a.  Processor  message  and  advisory  formats  shall  not  require  additional 
interpretation  by  the  operator,  such  as  table  lookups  and  references  to 
documentation,  with  the  exception  of  lengthy  diagnostic  procedures  to  be 
followed  by  the  operator  following  an  abnormal  condition. 

b.  No  computer  program  shall  generate  a  message  or  advisory  identical  to 
one  generated  by  the  OS  or  by  another  program. 

c.  Off-line  error  messages  shall  contain  as  a  minimum  the  following 
information: 

(1)  Time  error  was  detected. 

(2)  Textual  description  of  error  condition. 
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(3)  Required  operator  action  where  applicable. 

(4)  Contents  of  instruction  register  and  program  counter  at  time  of 

error. 

(5)  Identification  of  trig^-.'ing  module. 

to)  Computer  program  or  system  execution  status  following  the  error. 

On-line  error  messages  shall  contain  as  a  minimum  tne  information  in  Items 
( 1 ) ,  t d) ,  and  ( 3)  above . 

3. 3*8.0  Program  Coding  Conventions.  Software  shall  conform  to  required 
coding  conventions  stated  below. 

a.  Eacn  line  of  source  code  snail  contain  no  more  than  one  statement. 

b.  Source  code  shall  be  clearly  and  conspicuously  annotated  to  explain 
ail  inputs,  outputs,  branches,  and  otner  items  not  implicit  in  the  code  itself. 

c.  Names  of  operator  commands,  data  entries,  program  components, 
variables,  procedures,  and  other  software  components  shall  be  consistent  with 
tnose  used  in  system  design. 

d.  Code  snail  be  written  such  that  no  code  is  modified  during  execution. 

(M) 3. 3*8.7  Character  Set  Standards.  Character  sets  shall  conform  to 
standards  in  NBS-FIPS-PUB-l  Standard  Code  for  Information  Interchange,  ANS 
X3. 4-1978. 
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f  •>>"f  11-1-3.  Model  System  Specification  Section  Four 
*’•  vt'Al.iTY  ASScrirthCt  PROVISIONS: 

sec  tion  specifies  tr.e  requirements  for  formal  verification 
of  *  tv  irnir  ,  const  rue  arid  performance  of  tne  XXX  system  in  order  to 
:  -v.  •  e  cos.j  l  iar.ee  •.‘.tn  all  requirements  of  Section  3  of  tnis 
■  pc  i  f: .  alien.  to  or.  of  tr.e  requirements  of  Section  3.  union  is  to  be  verified 

Ly  r:.g:  :.-erir.g  test  a-.a  evaluation  is  specified  in  Table _ _  cy  its 

ie.ti.ir.  e.-.ts  paragraph  r. jitter.  Development  Test  and  Evaluation  (DT4E)  shall 
:e  «.v..aip  lisned  at  (  '_>  ».-nt_  or  contractor)  facilities.  Operations  Test  and 

Evaluation  i o r a. t  J  snail  be  accomplished  at  _ _ _ 

fa  'i  lines.  .  ’  . ~ 

a.1.1  fr  .  j  -  nti.ni  - 1  tv  for  Ven  fi  cation.  General  responsibility  relating  to 
test  and  evaluation  are  give;,  herein;  specific  details  are  given  in  the 
appropr iate  Statement  of  Work.  Tne  ESD  Program  Office  (PC)  will  be  the 
hespcs.sible  Test  0:  ga’’.  /.at  ion  RTO)  and  will  be  responsible  for  the  overall 
(r.j.iagt, merit  of  test  and  evaluation  of  tne  XXX  system.  Government 
rep  resent at i vets)  designed  by  tne  HTG  will  witness  all  formal  test  activities, 
unless  tne  contractor  is  notified  otherwise  by  the  PCO,  ana  will  certify  test 

C  ti  t.  d  . 

t.l.C  Gj.t  c  ial  Test  (Option^).  (Examples  of  special  tests  could  be 
eiv.r.  i  r.  t  a.  1 ,  S/V,  :  «*::.;.est ,  etc.) 

**•2  quality  Corifora.ar.ee  Inspection.  Requirements  for  formal  tests  of  system, 
functional  area,  and  C1/CPC1  performance  design  characteristi cs  and 
operability  as  defined  in  Section  3  shall  be  accomplished  as  specified  in  the 
XXX  verification  cross  reference  matrix.  The  explanation  of  the  XXX  test 
verification  cross  referer.ee  methods,  and  the  verification  tests  are  specified 
below. 

A.P.i  Verification  Cross  Reference  Matrix.  XXX's  verification  cross 

reference  matrix  is  contained  in  Table  _ .  The  verification  cross 

referer.ee  matrix  identifies  all  Section  3  requirements  by  paragraph  number  and 
title,  method (s)  of  verification,  and  test(s)  where  verification  shall  De 
performed.  Ine  verification  methods  are  indicated  by  the  notations:  I 
( inspection) ,  A  (analysis),  D  (demonstration)  and  T  (test).  The 
verification  tests  are  indicated  by  the  notation  P  (preliminary  qualification 
test / i  r,  forma  1  test),  F  formal  qualification  test),  S  (system  integration 
test)  and  0  (operational  test  and  evaluation).  All  tests  associated  with  a 
requirement  shall  be  listed  by  notation(s) ,  (P,  F,  S,  and/or  0)  under  the 
appropriate  metnod(s)  of  verification. 

A.ti.P  Methods  of  Verification.  Tne  definition  of  inspection,  analysis, 
demonstration,  and  test  are  defined  below. 

A.P.P.l  Inspection.  Verification  by  visual  examination  of  the  item, 
reviewing  descriptive  documentation,  and  comparing  the  appropriate 
characteristics  with  a  reference  standard  to  determine  conformance  to 
requirements.  Tnis  includes  mechanical  Inspection  of  equipment,  verification 
of  accuracy  and  completeness  of  documentation,  data  storage  table  structure 
and  capacity,  and  computer  program  source  code  audits. 


8 


HELPS,  Volume  11,  hevtslon  i  1  December  1983 

Contract  Management 


4. 2. 2. 2  Analysis.  Verification  by  evaluation  or  simulation  using 
mathematical  representations,  charts,  graphs,  circuit  diagrams,  or  data 
reduction.  lhts  includes  analysis  of  algorithms  independent  of  computer 
implementation,  analytical  conclusions  drawn  from  test  data,  and  extension  of 
test-produced  data  to  untested  conditions. 

4. 2. 2. 3  Deaoristrat  ton.  Verification  by  operation,  movement  or  adjustment  of 
the  1  teen  under  a  specific  condition  to  (>t-rform  the  desired  function.  This 
Includes  content  and  accuracy  of  displays,  comparison  of  system  products  with 
Independently  derived  test  cases,  primpt  system  recovery  f rom  ini  iced  failure 
conditions,  and  verification  of  reliability,  maintainability,  and  availability 

4. 2. 2. 4  Test, .  Verification  through  systematic  exercising  of  the  applicable 
item  under  all  appropriate  conditions  with  instrumentation  and  collection, 
analysis,  and  evaluation  of  quantitative  data.  This  includes  electrical 
continuity,  proper  operating  voltages,  correct  grounding  tolerance  of 
interference,  correct  computer  program  control  flow,  correct  computer  program 
data  flow  and  acceptance  of  proper  ranges  of  values. 

4.2.3  Verification  Tests.  Formal  verification  tests  snail  be  establisned  to 
assure  the  XXX  software  meets  all  requirements  allocated  to  software 
elements.  These  tests  snail  consist  of  Development  Test  and  Evaluation  tDT&E) 
and  Operational  Test  and  Evaluation  (OT&E). 

4.2. 3- 1  Development.  Test  and  Evaluation  tDT&B).  DT&E  is  accomplished  to 
verify  that  the  CPCI(s)  meets  performance  and  design  requirements.  DT&E 
consists  ol  PUTs,  KjTs,  and  an  SIT. 

4. 2. 3«1»1  CPCI  Preliminary  Qualification  Teats  (PQTs).  Tests  conducted  for 
tnose  functions  or  CPCts)  union  are  critical  to  the  CPCI.  The  selector  of 
critical  functions  to  be  tested  during  POT  may  be  based  on  time  or  performance 
critical  requirements. 

4.2. 3-  1.2  CPCI  Formal  Qua! i f i cat  ion  Tests  iFQTs).  Tests  accomplished  to 
verify  that  eacn  CPC  meets  tne  requirements  of  its  CPCI  specif -cation  and  that 
tne  requirements  of  3* 3*3  herein,  nave  been  met.  Program  and  group  programs 
shall  be  tested  to  verify  tnat  they  perform  their  intended  function  properly 
and  to  verify  tne  Interoperabi 1 ity  of  each  CPC  with  all  other  CPCs.  This 
process  shall  be  applied  to  the  individual  software  CPC  of  each  CPCI  and  shall 
be  continued  until  all  programs  have  been  verified. 

4. 2. 3. 1.3  System  Integration  Test  (SIT).  Test  conducted  to  provide  that  all 
CPCs  and  CPCKsl  wording  together  meet  all  system  performance  requirements. 

SIT  verifies  that  the  full  configured  XXX  performance  is  in  accordance  with 
requirements  of  the  specifications.  SIT  shall  be  performed  on  the  first 
article. 


4. 2. J.2  Operation  Test  and  Evaluation  (OT&E).  OT&E  determines  whether  the 
system  will  satisfactorily  perform  the  function  for  which  it  is  designed  in 
the  mission  environment. 
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Section  2.  Statement  of  Work 

2-1.  Review  ESD-TR-77-16,  SOW  Preparation  Guidebook. 

2-2.  Review  Figure  II-2-1,  Sow  Checklist  for  Computer  Resources. 

2-3.  Review  and  adapt  Figure  II -2-2,  Model  Computer  Program  Management  Task 
for  a  Full  Scale  Development  (FSD)  Statement  of  Work  (SOW). 

2-4.  Contact  ESD/OCH  for  use  of  the  automated  SOW  available  as  part  of  the 
Computer  Generated  Acquisition  Document  System  (CGADS). 

2-5.  Review  and  adapt  Figure  II-2-3,  Sample  SOW  Paragraph  for  ATLAS 
Requirement  if  software  for  support  equipment  is  to  be  developed  under  the 
contract . 
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Figure  Il-k-1.  SOW  Checklist  for  Computer  Resources 


No  ft. :  Tnose  items  succeeded  by  an  (M)  are  mandatory. 


-iM).  Has  tne  contractor  been  tasked  witn  preparing,  updating  and 
delivering  tne  Computer  Program  Development  Plan  (CPDP)  IAW  tne  DID?  (Arfi 
ouu-Ii,  Vol  II,  Para  i-9'i  (Note:  If  a  CPDP  was  required  as  part  of  IFPP,  SOW 
usually  requires  delivery  of  final  CPDP  and  as  required  updates).  Has  tne 
cor, tractor  been  tasked  to  adnere  to  tne  CPDP? 


f(M).  Has  tne  contractor  been  tasked  witn  adhering 
program  requirements  listed  in  Para  i.1.8  of  tne  system 


to  tne  computer 
specification? 


;j(M).  Has  tne  contractor  been  tasked  with  treating  firmware  as  software? 

i«Fh  Cub- it,  Vo  1  . ,  ArSC  Sup  1; 


■*.  uoes  tne  .),*  contain  requirements  similar  to  those  listed  in 
i.oe  aDf.w'a  Model  l.J«  ias».  for  software  development? 


Vv M ! .  has  tne  contractor  oeer  tasked  witn  tne  preparation  of  aeve-opment 
.  he  land  product  i  .:*•>)  specifications?  (AFP  800-11,  Vol  II,  Chapter  7  and 
.'D-161  Appendix  I  ill 

o  .  M  ' .  nas  tn-  contractor  ueeii  tasked  witn  Preliminary  and  Formal 
veal Lfication  Testing  and  system  Integration  Testing  of  each  Computer  Program 
,on! iguratiori  Item  in  accordance  witn  contractor  prepared,  AF  approved  test 
plans  and  procedures?  (NuTD:  PwT  responsibility  may  be  delegated  to  the 
contractor  and  not  specifically  identified  as  a  separate  test)  (AFP  SuC-li. 
Vo.  11,  Para  b- i  ana  AFh  60-11,) 

7iM;.  Have  tne  specific  Operational  Test  and  Evaluation  requirements  and 
contractor  responsibilities  been  clearly  defined?  (AFh  600-11,  Vol  II,  Para 


6(M).  Has  tne  contractor  been  tasked  witn  tne  development  of  a  Software 
wua.ity  Assurance  (SUA;  Program  in  accordance  witn  MIL-S-5877QA'.-  (AFSCR  7--I  - 

ViM).  Has  tne  contractor  been  tasked  witn  preparing,  delivering,  and 
updating  tne  SwA  Plan  or  been  tasked  witn  incorporating  tne  SQA  Plan,  in  tne 
CPDP  IAW  tne  DiDl  (Note:  Delivery  is  not  mandatory,  in  which  case  iAW  DID 
woui ”  r * o r  apped  r. / 

lolMj.  has  tne  contractor  oeen  tasked  witn  requesting  a  Computer-  Program 
identification  Number  iCi'lNj  for  eacn  CPC1  and  using  this  CP1N  along  with  tne 
specification  numoer  on  ail  documentation  pertaining  to  tne  GPCI  in  aooorair.ee 
witn  tne  DID.  (AFi<  600-1'*,  Voi  il,  Para  b-b)  i Dl-E-jitrA  should  appear  ir. 
tne  .urfi..  )  Also,  this  tasking  may  oe  in  a  conf  igura  t  i o  management  tasx.  : 

I..  Have  -tW  considerations  been  made  an  item  for  PMRs? 

If.  Are  preliminary  design  reviews  and  critical  design  reviews  IAW 
MiL-oTb-15FlA  required  for  all  CPCIs  with  agendas  and  minutes  prepared  by  tne 
contractor  and  delivered  iAW  the  DID?  (May  be  contained  in  tne  Keviews  and 
Audits  I  ask.  ) 
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13.  Has  software/computer  resources  been  included  as  a  possible  topic  for 
technical  intercha.-ige  meetings? 

14.  Are  functional  and  physical  configuration  Audits  IAW  MIL-STD-1521A 
required  for  each  CPCI?  (These  may  be  combined.  Also,  the  tasking  may  be  in 
a  reviews  and  audits  task.) 

15.  Is  the  contractor  tasked  with  preparing  IAW  MIL-STD-483  and 
delivering  IAW  the  DID  configuration  management  related  data  for  CPCIs?  (Eng. 
engineering  change  proposals  version  description  documents,  configuration 
index,  change  status  report  (computer  program),  specification  notice  (computer 
program).  Also,  this  tasking  may  be  in  a  conf iguration  management  task.) 

16.  Is  the  contractor  tasked  with  using  ATLAS  (Common  ATLAS  as  defined  in 
the  IEEE  Std  716-1982  or  its  approved  successor)  as  the  HQL  for  all  support 
equipment  software?  (See  Model  SOW  paragraph,  figure  II-2-3.) 
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Figure  11-2-2.  Model  Computer  Program  Management  Task  for  a  Full 
Scale  Development  (FSD)  Statement  of  Work  (SOW) 

The  following  is  a  model  Computer  Program  Management  Task  for  a  Full-Scale 
Development  Statement  of  Work.  This  model  task  doesn't  include  all  the  work 
the  contractor  has  to  do  in  relation  to  software.  So,  when  writing  the 
Computer  Program  Management  Task,  a  cross  check  of  the  other  tasks  will  be 
required  to  insure  all  the  work  is  specified.  The  following  is  a  list  of  some 
of  the  software  tasks  not  included  in  the  Computer  Program  Management  Task 
(also  listed  after  the  subject  is  where  the  task  is  usually  specified  and  the 
appropriate  wording  that  is  included). 

A.  Computer  Program  Identification  Numbers  (CPINs) 

(Usually  asked  for  in  the  Configuration  Management  Ta3k) 

The  contractor  shall  request  a  CPIN  for  eacn  CPCI  in  accordance  with  the 
CDRL.  The  contractor  shall  use  the  CPIN  along  with  the  specification  number 
on  all  documentation  pertaining  to  the  CPCI. 

6.  Reviews  and  Audits 

(Usually  asked  for  in  the  System  Engineering  Task) 

As  part  of  the  system  engineering  process,  the  contractor  shall  conduct 
the  following  contractual  reviews  IAW  MIL-STD-1521A. 

C.  Testing 

(Usually  asked  for  in  the  Test  and  Evaluation  Task) 


Figure  II-2-3.  Sample  SOW  Paragraph  for  ATLAS  Requirement 

NOTE:  Tnis  paragraph  should  be  included  in  the  SOW  when  software  for  support 
equipment  is  to  be  developed  under  the  contract. 

"All  test  programs  must  be  coded  in  ATLAS  (Common  ATLAS  as  defined  in  the 
Institute  of  Electrical  and  Electronics  Engineers  Std  IEEE  716-1982  or  its 
approved  successor).  Any  deviations  from  the  standard  must  have  prior 
approval  of  the  appropriate  authorities.  In  such  cases,  the  contractor  must 
fully  justify  and  document  all  such  deviations/extensions." 
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TASK  XX 

Computer  Program  Management 
X  X .  1  GEN ERAL  REQUIREMENTS 


The  contractor  shall  design  and  develop  the  computer  programs  to  satisfy 
the  design  and  performance  requirements  in  Specification  No.  XXX.  The 
contractor's  approach  to  software  development  under  this  contract  shall 
conform  to  that  approacn  presented  in  the  contractor-prepared,  government 
approved,  Computer  Program  Development  Plan  (CPDP)  which  shall  be  delivered 
and  updated  by  the  contractor  in  accordance  with  the  DID.  The  PCO’s  approval 
of  the  CPDP  snail  not  relieve  the  contractor  from  complying  with  any  of  the 
requirements  of  this  contract.  All  computer  programs  shall  be  controlled  in 
accordance  with  tne  requirements  established  in  the  Configuration  Management 
Task  of  tnis  SOW. 

The  overall  intent  of  this  task  and  the  computer  programming  requirements 
of  paragraph  3-3.8  of  Specification  No.  XXX  is  to  mandate  minimum  computer 
programming  requirements;  this  shall  not  preclude  exceeding  these  minimum 
requirements. 

X X . 2  Software  Development  Technologies/Management  Practices 

All  computer  programming  accomplished  under  this  contract  shall  be 
accomplished  in  accordance  with  the  management  practices,  tool s/teonniques , 
and  programming  and  organizational  structure  as  defined  in  tne  CPDP  and  the 
following  software  development  technologies  as  defined,  in  opecification  No. 
XXX,  paragrapn  3-3-8; 

a.  Top  Down  Design 

b.  Top  Down  Implementation 

c.  Use  of  Hign  Order  Language 

d.  Structured  Coding 

Tne  contractor 's  approacn  to  implementing  each  of  these  development 
technologies. management  practices  shall  be  defined  in  the  CPDP.  The 
approaches  described  m  the  CPDP  shall  become  effective  upon  contract  award 
and  snail  remain  in  effect  through  final  acceptance  of  all  project  software  bv 
tne  PCO.  Tne  Government  reserves  the  right  to  review  or  inspect  documenta".  ion 
or  facilities  which  verify  compliance  with  development  technologies 'management 
practices  defined  in  the  CPDP. 

XX. 3  LI  PE  CYCLE  ACTI V IT IPS 

XX. j. 1  Ana  lyses 

The  contractor  shall  analyze  the  requirements  of  Specification  No.  XXX  and 
accomplish  the  following  studies.  Those  mi!’,  be  prepared  and  delivered  I  AW 
the  ICf). 
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XX. 3* 1.1  Sizing  and  liming  Analysis 

The  contractor  snail  perform  a  detailed  sizing  and  timing  analysis  of  each 
CPCI.  The  contractor  snail  combine  the  results  of  these  analyses  to  determine 
the  composite  sizing  and  timing  requirements.  Tne  contractor  shall  also 
examine  in  detail  tne  dependence  of  the  sizing  and  timing  requirements  on  the 
overall  system  design.  [NOTE:  For  large  systems,  requiring  sizing  and  timing 
analysis  to  tne  CPL  level  may  be  more  appropriate.  If  the  referenced 
paragraphs  of  tne  system  specification  for  tne  system  being  addressed  do  not 
contain  information  on  software  utility  services,  then  the  last  sentence  of 
Paragraph  XX. 3.2.1  snould  not  be  used.] 

XX. 3*1*2  Data  Base  Architecture  Analysis 

The  contractor  snail  examine  tne  data  base  architecture  as  a  critical 
topic  in  software  design.  Alternate  approaches  to  data  base  management, 
addressing,  and  directory  structure  snail  be  investigated  by  the  contractor 
(in  terms  of  memory  requirements,  access  and  update  timing,  and  ease  of  data 
base  expansion)  in  order  to  determine  an  approacn  which  achieves  the  access, 
buffering  and  queueing  capabilities  required  to  meet  the  functional 
requirements. 

XX. 3*1. 3  Risk  Analysis 

The  contractor  snail  define  and  analyze  areas  of  software  risk. 

XX. 3*2  Design 

The  contractor  snail  document  the  rationale  supporting  his  system  design 
IAW  the  DID  as  follows:  | 

XX. 3*2.1  Software  Design  Criteria  and  Decisions 

Tne  contractor  shall  define  the  design  criteria  and  decisions  for  all 
software  by  graphic  representation  and  supportive  text  (i.e.,  software 
diagrams,  interface  definition,  trade  studies,  or  rationale).  The  contractor 
may  propose  to  develop  the  software  using  a  different  set  of  software  utility 
services  tnan  will  be  delivered  with  the  system.  In  this  case,  the  contractor 
shall  demonstrate  that  the  delivered  software  utility  services  meet  the 
operation,  maintenance,  and  training  capabilities  specified  in  Paragraphs 
3*1*7,  3*5.1,  and  3*6.2,  respectively,  of  Specification  No.  XXX  prior  to 
system  test.  [NOTE:  For  large  systems,  requiring  sizing  and  timing  analysis 
to  the  CPL  level  may  be  more  appropriate.  If  the  referenced  paragraphs  of  the 
system  specification  for  the  system  being  addressed  do  not  contain  information 
on  software  utility  services,  then  the  last  sentence  of  Paragraph  XX. 3. 2.1 
snould  not  be  used.] 

XX. 3.2.2  Hardware  Selection  Criteria  and  Decisions 

The  criteria  and  decisions  for  the  selection  of  all  computer  hardware 
shall  be  defined  and  delivered  to  the  Government  in  the  form  of  supportive 
text  IAW  the  DID.  | 
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[NOTE:  The  Information  can  be  included  in  tne  Data  Item  Descrioti j.i  ,DID)  for 
a  Technical  Report  (DI-S-3591A)  or  in  the  CPD!  DID  ( DI-S-30567A) .  In  either 
case,  modification  of  the  DID  is  required.] 

XX. 3-2. 3  CPCI  Organizatio;  and  Decomposition 

A  complete  list  and  the  rationale  behind  the  organization  of  CPCIs  shall 
be  defined  and  delivered  to  the  Government.  Appendix  XVII  of  MIL-STP-483 
shall  be  used  as  a  guide.  The  following  snail  apply  to  the  organization  of 
CPCIs: 

a.  The  number  of  CPCIs  shall  be  minimized. 

b.  CPCIs  shall  not  be  organized  across  vendor  product  lines.  That 
is,  a  CPCI  shall  operate  and  be  tested  on  a  single  vendor  computer  system. 

c.  A  separate  CPCI  shall  be  provided  for  each  unique  OS.  A  unique 
OS  is  defined  to  be  a  single  vendor  product  which  controls  the  allocation  of 
computer  resources  for  a  single  vendor  computer  system. 

d.  A  separate  CPCI  snail  be  provided  for  each  unique  set  of  software 
utility  services.  Unique  software  utility  services  are  defined  as  the  set  of 
software  utility  services  (compilers,  assemblers,  diagnostics,  and  editors) 
which  a  vendor  provides  to  support  a  single  computer  system. 

XX. 3. 3  Testing 

The  contractor  shall  test  the  project  software  in  accordance  with  Section 
4  of  Specification  No.  XXX,  the  CPDP,  and  the  Test  and  Evaluation  Task  of  this 
SOW  using  contractor  prepared,  government  approved  test  plans  and  procedures. 

The  contractor  shall  prepare  a  separate  Test  Plan,  Test  Procedures  and  Test 
Report  for  each  CPCI  IAW  the  DID. 

XX. 3*3.1  System  Integration  Testing 

The  contractor  shall  conduct  system  integration  testing  in  order  to 
demonstrate  that  the  total  system  meets  the  requirements  of  the  system 
specification.  The  contractor  shall  provide  all  necessary  software  and 
hardware  test  tools  to  demonstrate  these  requirements.  These  tools  shall  be 
reviewed  by  the  Government  prior  to  beginning  system  integration  testing. 

XX. 3-1*  Documentation 

In  accordance  with  paragraph  3.3-8  of  Specification  No.  XXX,  tne 
Configuration  Management  Task  of  this  SOW,  and  the  CDRL,  the  contractor  shall 
deliver  to  the  Government  the  documentation  for  design,  quality  assurance, 
delivery,  installation,  operation,  maintenance  support  and  all  other 
requirements  for  all  generated  Configuration  Items  (CIs),  and  Computer  Program 
Configuration  Items  (CPCIs).  Computer  program  specifications,  vendor  supplied 
operating  systems  and  software  utility  services  documentation,  user's  manuals, 
reference  manuals  and  maintenance  manuals  shall  be  prepared  and  delivered  in 
accordance  with  the  DID.  | 
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ffie  application  of  top  down  modular  design,  implementation  and  structured 
programming  concepts  in  tne  subject  contract  may  encourage  tne  use  of  new 
software  documentation  methods.  Therefore,  tne  contractor  may  descriDe 
documentation  techniques  in  tne  CPDP  wnicn  meet  or  exceed  tne  requirements  of 
present  B‘3  ana/or  Co  standards.  Government  review  and  approval  of  all 
alternate  t>  >  and/or  Cb  documentation  snail  be  accomplished  prior  to 
. mp .eme:  t  •  t ion. 

.~or  alj.  firmware  generated  i>y  tne  contractor,  documentation  of  tne 
computer  programs  and  data  snail  be  in  accordance  witn  MIL-STD-w83,  Appendix 
1  or  XV  i.  Tne  memory  portion  ana  the  hardware  portion  snail  be  documented  ir, 
accordance  with  MIL-STD-4Q0,  Appendixes  II  and  VIII. 

XX . 4  SUPPoHT  GurTWAhh 

Tne  term  "support  software"  refers  to  tne  collective  support  of  tne 
Operating  System  (OS)  and  software  utility  services  as  described  in  paragraph 
jo. 8  of  specification  No.  XXX. 

Tne  term  "of f-tne-sne If "  refers  to  an  item  wnicn  nas  been  produced  and 
placed  in  stock  by  a  contractor  prior  to  tne  contractor  receiving  orders  or 
contracts  for  tne  sale  of  tne  item,  in  addition,  tne  item  must  (1)  nave  been 
delivered  to  at  least  one  customer  (military/federal  or  commercial),  (2)  have 
passed  said  customer  acceptance  testing,  and  (3)  oe  operating  under  said 
customer  control  witnin  tne  user  environment.  Tne  contractor  may  produce  the 
item  to  eitner  commercial  or  military/federal  item  specifications  or 
descriptions.  Off-the-snei f  items  include  items  stocked  by  distributors  for 
wnicn  uovernment  contracts  may  be  received. 

Tne  contractor  snail  identify  all  off-the-sneif  support  software  necessary 
to  develop,  produce,  operate,  modify  and  maintain  the  project  sofware.  Tnis 
description  snail  be  available  for  Government  Review  and  Approval  and 
delivered  I AW  tne  bib. 

Tne  support  software  snail  ye  identified  in  tne  CPDr  and  shall  operate  on 
a  contractor  defined  computer  system,  wnicn  is  subject  to  Government 
approval..  All  support  software  shall  be  capaole  of  being  executed  using  a 
standard  off-the-sneif  operating  system  wnicn  meets  the  requirements  of 
paragraph  3. 3.8  of  specification  no.  XXX. 

XX. 5  SOFTWARE  DEVtL.uPMfc  NT  REVIEWS 


X X . b • 1  Software  Reviews 

Tne  contractor  snail  report  to  the  Government  on  tne  status  of  the  design, 
coding,  testing,  documentation,  and  maintenance  of  all  project  software  as 
part  of  tne  Program  Monthly  Reviews.  I'he  software  reviews  shall  address  tne 
contract  progress  of  tne  software  .•evelooment  activities  up  to  the  date  of  tne 
review  and  snail  include  visual  1  »n«-,tat ion  addressing  project  software  down 
to  tne  module  level  including  .:.;Lng  ami  timing  data.  In  addition,  the 
software  reviews  sho i .  inilude  a  review  of  the  3tatus  of  tne  management 
practices  and  tools/ toer.n ques  .g.d  urogram  organisational  structure. 
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XX. 5. 2  Ioformal  Software  Development  Reviews 

The  Government  reserves  tbe  right  to  inspect  any  information  or 
documentation  generated  by  the  contractor  and  subcontractor  personnel  assigned 
to  tnis  tasK  or  to  witness  any  test  associated  with  this  task  at  any  time 
wttnin  the  normal  work  schedule  of  suoh  personnel.  It  is  the  Intent  of  the 
Government  to  have  access  to  contractor  information/docuoentation  relating  to 
this  task  at  the  contractors  facility  without  interfering  with  the  contractor 
and  the  performance  of  tnis  task. 

XX. 5. 3  Formal  Reviews 

The  formal  review  procedures  as  described  in  the  System  Engineering  Task 
of  this  SOW  shall  be  applied  to  all  computer  programming  efforts. 

XX. 6  APPLICABLE  DOCUMENTS 

The  contractor  shall  comply  with  the  following  documents  to  the  extent 


specified : 

Number  and  Date 

Title 

Tailored 

Application 

spec  number 

Spec  date 

spec  title 

All 

MIL-STD-483 

31  December  1970 
Notice  2 

21  March  1979 

Configuration  Management 
Practices  for  Systems 
Eduipment,  Munitions  and 
Computer  Programs 

Sections  1-6 

App.  XVII,  XVI,  VI 

MIL-STD-D90 

30  October  1968 
Notice  1 

1  February  1969 

Specification 

Practices 

Sections  1-6 

App.  II,  VIII 
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■action  3«  Instructions  for  Preparation  of  Proposal  (IRpp) 

3-1.  Insure  tnat  tne  IFPP  la  consistent  with  toe  Advanced  Procurement  Plan, 
the  Source  Selection  Plan,  -no  particularly  the  evaluation  criteria. 

3-2.  Review  Figure  II-3-1,  IFPP  Checklist  for  Computer  Resources. 

3-3.  Review  Figure  II-3-2,  Model  Data  Rights  Requirement  for  IFPP. 

3-1.  Review  Figure  11-3-1,  Model  Computer  Resource  Requirements  for 
Management  Volume  of  IFPP. 
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Figure  II-3-1,  IFPP  Checklist  for  Computer  Resources 
NOTE:  Items  succeeded  by  an  (M)  are  mandatory. 

1.  Does  ttie  IFPP  contain  ioputs  similar  to  those  listed  In  ESD/ALEQ's 
model  IFPP  paragraph? 

2.  Is  the  proposer  required  to  submit  a  preliminary  Computer  Program 
Development  Plan  (CPDP)  and  Software  Quality  Assurance  Plan  (SQAP)  with  his 
Droposal?  (SQAP  may  be  iocluded  as  part  of  the  CPDP  per  AFR  74-1.)  if  not, 
is  software  specifically  required  to  be  covered  in  the  tecnnical  and 
management  volumes  of  tne  proposal? 

3.  Is  tne  proposer  required  to  address  tne  alternatives  listed  below? 

(a)  Lease  vs  purchase  of  equipment  and  programs. 

(b)  Computer  modularity  from  a  growth  vs  cost  and  schedule  standpoint 

(c)  Computer  program  modularity  from  a  maintainability  vs  cost  and 
schedule  standpoint. 

(d)  Proposed  use  of  existing  software. 

4(M).  Is  the  proposer  required  to  include  waiver  information  with  his 
proposal  if  he  plans  to  use  a  non-standard  HOL?  (AFR  300-10  including  AFSC 
Sup  1  and  AFSC  Sup  1  to  AFR  800-14,  Vol  I) 

5.  Is  the  proposer  required  to  address  areas  of  risk  in  the  development 
of  computer  hardware  and  software  with  regard  to  schedule  and  cost  impact? 

6.  Do  the  selection  criteria  include  management  of  computer  resources  and 
software  development  methodology? 
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Figure  II-3-2.  Model  Data  Rignts  Agreement 

NOTE:  Tnese  paragraphs  snouid  be  tailored  a3  appropriate  and  can  be  required 
in  almost  any  volume  of  tne  proposal  depending  on  tne  topic  to  be  stressed 
(e.g. ,  cost,  tecnnical,  logistics). 

X.X.X  INSTRUCTIONS  PERTAINING  TO  SPECIAL  DATA  PROVISIONS: 

Offerors  snail  submit  a  Summary  Letter  on  Data  and  Software/Firmware 
Rignts  setting  fortn  in  full  tneir  proposal  (including  all  subcontractor s ' 
positions)  regarding  sucn  technical  data  and  computer  software/firmware 
rignts.  Inis  Summary  Letter  snail  take  precedence  over  any  other  statement  in 
tne  offeror's  proposal  witn  regard  to  technical  data  and  computer 
software/firmware  rights.  Written  guarantees  from  subcontractors  (at  all 
tiers)  setting  fortn  tneir  position  on  data  and  software/ firmware  rights  snail 
also  oe  included  as  attacnments  to  the  Summary  Letter  including  tneir  express 
agreement  to  accept  tne  requirement  of  DAR  7-lC4.9(a)  tnat  it  be  flowed  down 
to  all  subcontracts.  The  Summary  Letter  and  the  subcontractor  guarantees 
snail  separately  address  each  subcontractor ' s  position  regarding  technical 
data  and  computer  software/f irmware  rights  in  regard  to  (a)  follow-on 
competitive  operation  and  maintenance  service  contracts  and  (b)  follow-on 
competitive  production  contracts,  as  described  below. 

(a)  It  is  tne  requirement  of  the  Government  to  nave  tne  opportunity  to 
compete  follow-on  O&M  Service  contracts.  It  is  therefore  required  tnat  all 
tecnnical  data  and  computer  sof tware/f irmware  to  be  furnished  under  tnis 
solicitation  oe  furnisned  with  tne  rignts  as  set  forth  in  the  Rights  in 
Technical  Data  and  Computer  Software  clause  DAR  7-104. 9(a).  Specifically, 
note  tnat  subparagraph  (b)(l)(vii)  of  the  clause  requires  that  all  manuals  or 
instructional  materials  prepared  or  required  to  be  delivered  under  this 
contract  or  any  subcontract  hereunder  for  installation,  operation,  maintenance 
or  training  purposes  snail  be  provided  to  the  Government  with  unlimited  rights. 

(b)  Tne  Government  additionally  requires  the  ability  to  competitively 
procure  any  follow-on  production  or  Operation  and  Maintenance  (0&M) 
requirements  for  tne  XXX  which  may  be  ultimately  produced  after  Phase  XXX  of 
the  contract  contemplated  by  tnis  solicitation.  Therefore,  it  is  necessary  to 
obtain  a  tecnnical  data  and  computer  software/f irmware  package  suitable  for 
use  in  competitive  reprocurement  of  the  system  and  its  components  and 
competitive  procurement  of  04M.  For  this  reason,  it  is  required  that  the 
offerors  and  their  subcontractor s  submit  to  the  PCO  for  approval  a  plan  for 
avoiding  tne  use  of  any  restricted  rights  computer  s oftware/f irmware  or  items, 
components,  software/firmware  and  processes  for  which  technical  cata  would 
qualify  for  limited  rignts  under  the  criteria  of  DAR  7-104. 9(a)  such  tnat  the 
rights  provided  the  Government  would  be  inadequate  for  such  competitive 
follow-on  procurement.  In  addition  to  the  minimum  restricted  rights  in 
computer  software/firmware  aod  limited  rignts  in  technical  data  pursuant  to 
DAR  7-104. 9(a),  the  Government  requires  as  a  minimum  the  additional  right  to 
disclose  and  have  used  by  third  parties  any  such  technical  data  or  computer 
software/firmware  requlrea  under  the  cootract  including  Phase  XXX  CLIN  YYYY 
"Option  for  Additional  Technical  Data  and  Computer  Software",  for  the  purposes 
of  competitive  procurement  of  tne  XXX  system.  Where  necessary,  these 
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additional  rights  shall  be  acouired  by  the  Government  under  Special  Provision 
Z'L ,  "Option  for  Government  to  Acouire  Additional  Rights  in  Technical  Data  and 
Computer  Software,  Direct  Licensing  or  Technical  Additional  Subject,  to 
Equitable  Adjustment  in  Price"  and  CLIN  YYYY  "Option  for  Additional  Technical 
Data  and  Computer  Software". 
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Figure  II-3-3-  Model  Computer  Resource  Reoulrements  for  Management  Volume  of 
IFPP 

NOTE:  If  tne  contractor  Is  asged  to  deliver  a  preliminary  CPDP  with  his 
proposal,  the  following  moael  is  not  needed. 

X.X  Computer  Software  Development/Management 

X.X.i  Organizations,  Scneaules  and  Resources 

a.  Organizations 

(1)  Snow  tne  prime  and  subcontractor ' s  technical  organization 
structure,  responsibilities,  interfaces,  sjcill  requirements ,  ana  lines  of 
communication  for  his  program  office  for  tne  subject  contract. 

(2)  Show  tne  organizational  responsibility  ana  structure  of  tne 
software  development  office(s),  both  prime  and  subcontractor. 

b.  Deliverables/Dooumentatlon 

(1)  Describe  tne  approach  for  developing  computer  program 
documentation  to  include  preliminary,  interim  and  final  versions. 

(2)  Describe  any  new  software  documentation  tools/tecnniaues  wnicn 
meet  or  exceed  the  requirements  of  present  B5  and/or  C5  standards  wnicn  you 
intend  to  use  on  tne  subject  contract.  Tnis  description  snail  identify  tne 
specific  alternatives  by  describing  tne  necessary  changes  to  the  content  and 
format  of  tne  B5  and/or  C5  standards  as  described  in  MIL-STD-433.  Any 
proposed  changes  shall  at  a  minimum  describe  the  equivalency  features  between 
existing  documentation  requirements  of  the  standard  B5  and/or  C5  and  the 
proposed  a !ternative( s) .  Discuss  the  relationship  of  the  documentation  to  top 
down  design  and  structured  programming. 

(3)  Describe  the  approach  to  computer  program  in-line  commenting. 

(*0  Describe  your  methodology  for  choosing  CPCIs. 

c.  Progress /Status  Reporting 

(1)  Describe  your  approach  to  software  development  program 

reporting. 

(2)  Describe  your  approach  to  identifying  and  surfacing  software 
development  problems. 

(3)  Describe  tne  methods  and  procedures  for  collecting,  analyzing, 
monitoring,  and  reporting  on  the  size  and  timing  of  CPCIs. 

(M)  Describe  tne  conduot  of  all  Internal  project  reviews. 
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d.  Facilities 

(1)  Identify  software  development  facilities  to  be  used. 

(2)  Identify  all  computer  resources  including  simulators, 
prototypes,  etc.,  to  be  used. 

(3)  Identify  all  testing  resources  to  be  u-<ed. 

(4)  Identify  all  tools,  deliverable  and  not  deliverable,  that  are 
planned  for  use  in  the  development  of  the  testing  of  the  computer  programs. 

e.  Software  Design  Production  Schedule 

(1)  Detailed  software  schedule.  This  schedule  will  be  a  graphic 
representation  of  the  detail  development  milestones,  scheduled  activities  and 
deliverables.  The  grapnio  network  shall  include  an  identification  of  critical 
path  elements.  The  contractor's  schedule  3hall  identify  the  time  allocated 
for  the  Government's  review  of  software  and  hardware  related  data  submissions. 

(2)  A  list  of  delivery  dates  for  all  operational  and  support 
hardware,  operational  and  support  software,  and  each  data  submission  by 
documeot  number /title  will  be  included. 

f .  Labor  Pnasing 

(1)  Provide  a  proposed  schedule  for  labor  phasing  by  labor 
category  per  month  that  will  be  allooated  to  each  CPCI.  Table  1  shall  be 
completed  for  tne  composite  expenditure  Of  manpower  allocated  for  each  CPCI. 
Additionally,  a  definition  of  labor  categories  will  be  provided. 

(2)  Describe  the  positions  and  the  experience  level  of  personnel 
to  be  used  in  each  phase  of  the  software  development  cycle. 

(3)  Identify  any  key  personnel  in  the  software  development  area, 
their  anticipated  position,  their  qualifications  and  their  responsibilities. 

X.X.2  Software  Development  and  Testing  Methodology.  This  description  shall 
identify  the  contractor's  experience  with  each  of  the  software  development  and 
testing  methodologies  to  be  applied  to  the  subject  contract.  This  description 
shaLl  distinguish  between  existing  contractor  capabilities  and  those 
capabilities  to  be  implemented  solely  for  the  subject  contract. 

a.  Software  Requirements 

Tne  or far or  Is  required  to  use  an  approved  standard  High  Order  Language  (HOL) 
for  the  system  in  accordance  with  System  Specification  X.X.X.  If  use  of  an 
HOL  is  not  feasible,  a  Justification  for  use  of  an  alternate  language  must  be 
submitted  containing  information  as  indicated  in  (?)  below. 

(1)  Discuss  the  effects  of  the  proposed  language(s)  on  top  down 
design,  structured  programming,  data  base  design,  manpower  and  software 
maintenance.  Provide  a  dencription  of  each  language  translator  including  its 
.eve"  -?f  optimization  and  wro  capabilities.  The  offeror  shall  eussicer  the 
abeve  .  n  ot  .'-'•ect’.on  t  be'  ow . 
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\2)  Discuss  your  approach  to  implementing  the  retirement  for 
using  a  single  approved  HOL.  If  the  offeror  feels  that  the  software  must  be 
coded  in  a  noo-standard  HOL  or  assembly  language,  identify  the  language 
proposed  and  demonstrate  that  this  selection  will  satisfy  requirements  more 
effectively.  This  tradeoff  aoalysis  should  assess  programming  language 
selection  effect  on  manpower,  sohedule,  system  performance  and  software 
maintenance. 


(3)  Describe  your  approach  to  analyzing  system  requirements  ana 
deriving  tne  detailed  functional  requirements.  Show  the  mapping  of  functional 
requirements  to  CPCIs. 

(4)  Describe  tne  software  risk  areas  and  the  approacn  to 
minimizing  these  risks. 

b.  Design 

(1)  Describe  top  down  design  metnodologies  to  be  used. 

(2)  Describe  your  approach  to  data  base  design. 

(3)  Describe  your  plan  for  insuring  program  growth,  modularity  and 
ease  of  modification.  Also  include  description  of  the  plan  for  controlling 
computer  program  growth,  to  keep  it  within  any  imposed  limits. 

(4)  Show  the  estimated  size  in  each  language  to  be  used  for  each 

CPCI. 

(5)  Describe  all  existing  software  to  be  used,  its  laoguage,  and 
the  CPCI  it  will  be  included  in;  also  include  the  instruction  count  in  Table  1 
for  each  CPCI. 

(6)  Describe  all  modifications  to  existing  software,  if  any,  and 
include  instruction  counts  in  Table  1  for  each  CPCI. 

(7)  Specifically  identify  all  restricted  rignts  software  to  be 
delivered  aod  describe  tne  restricted  rights. 

c.  Code,  Debug  and  Unit  Test 

(1)  Describe  your  software  Implementation  approach. 

(2)  Describe  all  software  programming  languages,  practices, 
standards,  methods  aod  conventions  intended  for  use  00  this  cootract. 

(3)  Describe  all  software  development  documentation  and  the 
procedures  which  will  be  used  to  keep  the  documents  curreot. 

(4)  Describe  any  progress  monitoriog/reviews. 

(5)  Describe  any  computer  program  problem  reporting  methods. 

(6)  Describe  your  approach  to  unit  level  oode  testing. 
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0.  Development  Testing 

(1)  Describe  the  extent  of  In-plant  testing  Including  test 
planning,  test  methods,  documentation  and  test  reporting.  The  general 
procedures  for  reporting,  monitoring  and  resolving  program  errors  and 
deficiencies  discovered  during  lo-plant  testing  should  be  also  described. 

(2)  Describe  the  method  for  insuring  satisfactory  program  testing 
and  qualification  of  computer  programs.  This  should  include  a  discussion  of 
tne  approach  to  Preliminary  and  Formal  Qualification  Tests. 

X.x.3  Software  Management  and  Control 

a.  Design  and  Development  Controls 

Describe  your  software  management  practices,  tools,  aod  techniques 
and  how  these  will  control  the  risk  areas. 

b.  Configuration  Control 

(1)  Describe  facilities  and  procedures  to  be  used  to  maintain 
configuration  cootrol  during  the  software  development  to  include  error 
correction  cootrol  during  integration  aod  test,  as  well  as  development. 

(2)  Describe  facilities  and  procedures  to  be  used  after  first 
system  acceptance,  if  applicable. 

c.  Maintenance 

(1)  Describe  methods  for  in-plant  and  on-site  identification  and 
correction  of  software  and  documentation  deficiencies. 

(2)  Describe  the  maintenance  approaoh  for  all  vendor  supplied 
off-the-shelf  software  (operating  systems  aod  utility  services). 

(3)  Describe  the  maintenance  approach  for  all  contractor-developed 
software.  Include  in  this  description  how  you  conceive  the  distribution  and 
Installation  of  patches  and  other  changes  to  the  fielded  systems. 

(il)  Describe  the  capabilities,  tools,  and  equipment  to  be 
delivered  for  software  maintenance. 


Table  1.  General  Purpose  Software  Development  Worksheet 
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*1  For  language  use  F  for  Fortran,  A  for  Assembly  and  specify  any  others 
*2  Existing  _  An  instruction  requiring  no  alternation. 

*3  Conversion  An  instruction  requiring  an  alteration. 
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Section  4.  Contract  Data  Requirements  List  (CDRL) 


1.  Review: 

AFR  310-1 

AFSCR  310-1  including  ESD  Sup  1 

AFR  80-45  and  AFSC,  ESD  Supplements  to  it. 

Acquisition  Management  Systems  &  Data  Requirements  Control  List  (DOD 
5000. 19-L  Vol  II) 

2.  Review  Figure  II-4-1,  Table  of  Software-Related  DIDs  commonly  used  at  F.SD, 
for  applicability.  Modify  any  DID  necessary  to  meet  specific  requirements  and 
provide  the  necessary  back-up  sheets. 

3.  Prepare  CDRL  inputs  using  Figure  11-4-2,  Computer  Resources  CDRL  Checklist 

and  provide  to  Data  Manager. 

4.  Insure  that  every  deliverable  needed  is  contained  in  the  CDRL. 

5.  Contact  ESD/OCH  for  use  of  the  automated  CDRL  available  as  part  of  the 
Computer  Generated  Acquisition  Document  System  (CGADS). 


NOTE:  Data  is  expensive,  buying  data  without  a  valid  requirement  is  | 

wasteful,  not  only  of  government  money  but  contractor  time.  All  requirements 
for  data  should  be  reviewed  carefully  and  any  redundant  or  unnecessary  data 
eliminated.  Secondly,  another  way  to  reduce  the  high  cost  overhead  in 
purchasing  data  is  to  tailor  "down"  the  information  required  by  the  data 
item.  Thi3  requires  reviewing  Block  10  (Preparation  Instructions)  of  the  DD 
Form  1664  to  determine  if  all  the  information  the  contractor  has  to  provide  is 
needed.  If  not,  those  particular  paragraphs  of  data  snould  oe  deleted.  This 
Is  done  by  making  the  following  changes  to  the  form  (e.g.,  DD  1423,  AFSC  Form 
708)  used  in  the  CDRL:  annotate  the  data  item  number  with  a  "/T"  and  state 
what  paragraphs  are  deleted  in  the  remarks  section.  A  contract  Data 
Requirement  List  (CDRL)  should  reflect  data  which  is  useful,  important,  ar.d 
above  all,  necessary. 

NOTE  2:  There  are  three  different  kinds  of  unique  DIDs 

a.  Unique  DIDs  from  other  commands  -  still  valid 

b.  Unique  DIDs  for  general  ESD  use  -  not  valid  unless  stated  in  Ira 
Reimer  letter,  see  attached  letter 

c.  Unique  DIDs  for  a  specific  program  or  project  (The  Approval  Limitation 
block  would  specify  if  the  DID  applied  to  a  particular  program  or  contract, 
e.g.,  For  use  on  contract  F19C-00-1143)  -  valid  for  that  particular  program  or 
contract  and  any  follow-on. 
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Figure  1 1—4—2.  Computer  Resources  CDRL  Checklist 

1.  Insure  adequate  software  documentation  is  required  for  the  particular 

system  being  procured. 

2.  Check  for  consistency  of  requirements  of  the  CDRL  entries  with  the  SUW. 
Every  CDRL  item  must  be  tied  to  a  specific  SOW  task.  Ensure  reference  is  to 

correct  SOW  paragraph. 

3.  Check  delivery  dates  of  software  related  DIDs  and  make  sure  they  are 
reasonable  (e.g.,  Part  I  Specifications  delivered  about  60  days  prior  to 
Preliminary  Design  Review  (PDR) ,  Partial  Draft  Part  II  Specifications 
delivered  about  60  days  prior  to  Critical  Design  Review  (CDR),  Computer 
Program  Identification  Numbers  requested  prior  to  CDR,  Draft  CPCI  Test  Plan 
submitted  prior  to  CDR,  and  CPCI  Test  Procedures  submitted  after  Government 
approval  of  Test  Plan). 

*J.(M)  As  a  minimum,  have  the  following  Data  Item  Descriptions  (DIDs)  been 

included  in  the  CDRL: 

a.  Appropriate  DIDs  for  computer  program  specifications  (e.g.,  Part  I 
and  Part  II,  B  and  C,  and/or  non-complex  CPCI  Specifications) 
(DI-E-3119B  and  DI-E-3120B  most  common) 

b.  Computer  Program  Development  Plan  (CPDP)  (usually  DI-S-30567A) 

c.  Software  Quality  Assurance  Plan  if  a  separately  deliverable  item 
(may  be  included  in  CPDP  or  overall  QA  Plan)  (usually  DI-R-3521) 

d.  Actual  CPCI/Software  end  items  (e.g.,  DI-E-BOIRS) 

e.  Version  Description  Document  (DI-E-3121) 

f.  Specification  Change  Notice  for  computer  programs  (DI-E-3131*) 

g.  Computer  Program  Identification  Number  (CPIN)  Request  (DI-E-3162A) 

h.  CPCI  Test  Plans,  Procedures,  Reports  (OT-DI-E-30162,  OT-DI-E-30153, 
and  0T-DI-E-30154  are  recommended). 
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.lection  8.  Development  of  Source  Selection  Plan  (SSP) 

•■-I.  Review  otner  S3 Ps  for  guidance  in  preparation  (contact  tfte  ESD  Scurcr 
odection  Secretariat  at  Waltnam). 

■>-<?.  Review  tne  Source  Selection  Secretariat  Handbook  which  explains  the 
source  Selection  Process. 


b-f.  rteview  APR  7  u  - 1 1>  Source  Selection  Policy  and  Procedures  (Para  2.2)  which 
describes  SSP  content,  preparation,  and  minimum  requirements. 


•?-4.  Assist  with  draft  SSP  and  insure  it  contains  the  approach  and  Government 
organization,  plus  criteria  ana  schedule  for  proposal  evaluation  and 
contractor  selection. 


insure  draft  complies  witn  PMP  directions  regarding  source  selection. 
Coordinate  draft  with  Ssbb  membership. 

•.mu ruinate  final  SoP  for  approval  t»y  Source  Selection  Authority  vSSAi. 
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Sectloo  6.  Evaluation  Criteria 

6-1.  Review  APR  70-15  (Source  Selection  Policy  and  Procedures). 

6-2.  Review  Source  Selection  Plan  for  identification  of  the  evaluation 

eri'.  -'la. 

6-3*  losure  software  evaluation  criteria  covers  all  significant  reauirements 
in  the  Sygtem  Specification,  SOW  <4  CDRL. 

Are  all  software  criteria  Known  to  the  contractors?  Insure  proposals 
will  not  be  evaluated  against  criteria  unknown  to  the  contractors. 

6-5 •  Insure  each  criterion  includes  a  reference  to  tne  corresponding 
•equir aments  in  the  RF?  package. 

6-6.  Insure  Standards  of  Evaluation  are  also  prepared  for  evaluating  the 

criteria. 

6-7.  Are  tne  software  criteria  detailed  enough  to  require  a  "how"  answer 
instead  cf  a  restatement  of  the  requirement  by  the  contractor? 

6-8.  Has  a  risk  analysis  of  the  software  requirements  been  performed?  Will 
it  be  used  in  guiding  the  evaluation  process? 

6-9*  Are  software  criteria  written  to  evaluate  the  contractor's  abilities  to 
meet  the  contract  requirements? 

6-10.  Do  software  criteria  exist  that  will  discriminate  between  the  best 
contractor s? 

6-11.  Are  software  cost  criteria  realistic?  Are  they  based  on  reliable 
estimates? 

6-12.  Insure  the  complete  set  of  management  evaluation  criteria  covers  all 
management  information  required  by  the  CPDP. 


6—1 3 .  Review  Figure  II-6-1,  Sample  Computer  Resources  Evaluation  Criteria 
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Figure  Il-o-i.  Sample  Computer  Resources  Evaluation  Criteria 

i'r.e  fol_owing  evaluation  criteria  were  taken  from  a  sampling  of  actual 
evaluation  criteria  used  at  ESC.  Twenty-three  catagories  of  evaluation 
criteria  are  giver.. 

A.  Conf iteration  Management 

1.  is  configuration  Management  properly  coordinated  with  SQA? 

2.  Is  software  configuration  management  a  part  of  Software  Quality 
Control? 

3.  Is  _iorary  Control  properly  coordinated  woth  Configuration  Management? 

4.  Does  tne  company  provide  a  list  of  authorized  signatures  for 
configuration  control? 

b.  Docs  conf iguration  status  accounting  comply  with  MIL-STD-483? 

6.  Are  configuration  status  reports  provided  on  a  regular  basis? 

7.  Does  the  company  exhibit  a  written  Software  Configuration  Management 

Policy? 

0.  Is  tnis  policy  applied  to  all  software  projects? 

9.  Are  software  representatives  present  on  hardware  CM  boards? 

10.  Does  tne  contractor  provide  for  government  representation  on  CM  boards 

uu.  Are  hardware  representatives  present  on  software  configuration  boards? 

12.  Does  tne  company  present  a  sample  Configuration  Management  plan? 

13-  Is  a  form  provided  for  software  change  proposals? 

14.  I3  u  form  provided  for  software  trouble  reports? 

15.  Is  a  form  provided  for  software  change  reports? 

16.  Is  a  form  provided  for  software  trouble  analysis? 

^7.  Is  a  matched  document  tracking  procedure  used? 

IS.  Are  configuration  status  reports  submitted  on  a  regular  basis? 

19.  Are  software  change  approval  forms  used? 

20.  Does  each  required  CM  board  meet  on  a  regular  schedule? 

21.  Is  the  configuration  management  office  properly  staffed? 

22.  Are  EAM/MIS  techniques  utilized  for  configuration  status  accounting? 

23.  Is  software  configuration  authentication  properly  provided  for? 

2 4.  Are  provisions  made  for  adequately  numbering  and  identifying  software 
products? 

25.  Are  inter-relationships  between  software  and  hardware  CM  clearly 
defined? 

26.  Dots  the  company  adequately  provide  for  the  assurance  that  approved 
changes  are  implemented? 

27.  Does  the  configuration  procedure  adequately  describe  interface 
management? 

26.  Does  tne  company  exhibit  evidence  of  an  ongoing  software  configuration 
control  program  working  successfully  on  previous  projects? 

B.  Software  Quality  Assurance/Software  Quality  Control 

1.  Does  the  company  present  a  written  SQA  Policy? 

2.  Does  the  company  present  an  SQA  plan  which  they  would  apply  to  this 
project  as  part  of  tne  CPDP? 

3.  Can  tne  company  present  evidence  that  their  QA  is  working  successfully 
on  other  projects? 

4.  Does  the  SQA  plan  clearly  delineate  levels  of  authority  and  show  key 
contact  personnel? 
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5.  Does  the  company  exploit  a  Library  Control  Doctrine  and/or  Library 
'ops’  manual? 

6.  Are  tne  two  types  of  reauired  software  library  clearly  distinguished? 

7.  Are  Master  Programs  always  initiated  on  new  certified  storage  media? 

8.  Are  'working  media'  kept  separate  from  Master  baseline  files? 

9.  Are  library  transaction  records  maintained? 

.0.  Are  library  personnel  auaiified  as  librarians? 

11.  Does  the  company  have  a  procedure  for  work  certification? 

12.  Does  the  company  present  a  written  testing  policy  and  doctrine  for 
software? 

13.  Does  the  test  engineer  function  as  part  of  the  design  team? 

19.  Is  tne  test  engineer  a  'third  party  independent  functionary'  of  the 
S'jA  department  working  with  the  design  team? 

15.  Does  the  company  have  an  assigned  group  which  does  software  trend 
analysis? 

16.  Does  the  comoany  adhere  to  tne  ’Top  Down'  testing  approach? 

17.  Does  the  company's  formal  review  process  adequately  adCress  the  S/W 
issues? 

18.  Is  tne  same  level  of  control  (SQA)  exercised  over  ' non-deliverable'  as 
'deliverable'  SW? 

C.  Software  Development 

1.  .Are  the  contractor's  support  software  and  tools  (compilers,  etc.) 
adequate  for  this  oroiect? 

2.  Will  the  contractor  give  the  Government  full  license  for  all  support 
of  software  used? 

3.  Is  tne  contractor's  support  software  under  N'avy  Configuration 
Management? 

4.  Does  the  contractor  have  a  software  'specification  writing' 
depirtmect/cJ  vision? 

5.  Is  the  company  experienced  in  writing  Military  S/W  spec,  for  embedded 
computer  3 ? 

6.  Is  a  qualified  mathematician  operating  as  part  of  the  spec  group? 

7.  Does  the  company  advocate  the  MIL-STD-490  Type  'A'  approach  to 
specification  writing? 

8.  Does  the  company  propose  to  develop  Interface  design  specifications  in 
the  initial  specs? 

9.  Is  tne  specification  plan  phase  properly  defined  in  the  SD?  proposal? 

10.  Is  the  'Top  Down'  approach  to  specification  writing  advocated 
practiced? 

11.  Is  the  design  approach  proposed  a  trua  Top  Down  Hierarchical 
'approach'? 

12.  Is  a  ’tree  structure'  design  proposed  for  the  PDS? 

13.  Are  illegal  desigo  structures  proposed? 

iu.  Does  the  company  utilize  the  'lead  programmer  team'  principle  for 
designing  software? 

16.  Is  the  Lead  Teem  Programmer  assigned  to  write  the  Program  Design 
Specification? 

16.  Is  the  Assistant  Lead  Team  Programmer  assigned  to  wr^te  the  Data  Base 

Design? 

17.  Is  ;ne  Lead  Programmer  fully  oualified? 

18.  is  ‘he  team  ocieau- tely  staffed  to  perform  efficiently? 
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i  j  .  It  vr.e  team  greater  tnan  eignt  persona'.- 

20.  I:  tr.e  team  too  small? 

21.  Are  team  cnaonela  of  oop>muoicatic.i.s  adequate. y  defined? 

22.  It  team  memoer  wor :.-scneauling  adequately  defined  and  documented? 

2j.  Dees  team  ooctrine  define  procedures  for  updating  and  correcting  team 
documental  *  in' 

2a.  It  one  Deaa  Programmer  a  part  of  tne  CCB? 

25.  Does  the  team  report  to  tne  proper  level  of  authority? 

26.  Dc  .-s  team  organization  aaeouately  provide  for  wor*  scheduling  and 
moo -taring'. 

2?.  Are  team  activities  and  functions  properly  coordinated  with  the  SCA 
division? 

26.  Do c 3  i.'it-  team  (or  each  team)  have  team  operations  manuals? 

29-  Dc  beams  operate  continuously  on  a  numoer  of  scheduled  projects? 

30.  Arc  teams  chosen  and  selected  as  each  project  Is  developed? 

31.  Does  tne  company  prescribe  a  well  formatted  and  defined  project 
note dc ok? 

31.  Are  projecc.  notebooks  Kept  by  tne  Dead  Programmer? 

33.  Can  tne  company  snow  examples  of  project  noteoooks? 

jc.  Are  pr ogrammer a  notebooks  Kept? 

3;..  Can  tne  company  snow  examples  of  'programmers'  notebooKs? 

3c.  Are  menial  tasks  performed  by  tne  appropriate  skill  levels? 

; 7 .  Dc  software  designers  do  tneir  own  coding  ana  keypunching? 

3o.  is  coding  done  by  team  members  from  a  pool  of  coders.  Coot  team 
members)? 

39-  Does  tne  company  clearly  indioate  tnat  coding  will  begin  at  the  proper 
point  in  tne  development  cycle? 

90.  Does  tne  company  call  for  use  of  FDD  techniques? 

41.  It.  the  data  base  properly  defined  before  module  design  begins? 

42.  Are  data  Dase  designs  sufficiently  complete  before  module  design 
begins? 

43.  Are  all  moaule  designs  submitted  to  the  structured  walk-througn 

process? 

44.  Are  structured  walk-throughs  adequately  utilized? 

43*  W;.l  the  contractor  allow  the  Government  to  view  tne  structured 
walk- thrown? 

4n.  Ar>:  mmutes  kept  of  the  structured  walk-throughs? 

47.  Arc  user  manuals  developed  concurrently  with  system  designs? 

46.  Are  test  plans  initiated  concurrently  with  program  designs? 

49.  Is  tne  test  engineer  a  member  of  the  SQA  department  working  with  the 
team? 

D.  Facilities  Evaluation 

1.  0r«3  tne  company  own  and  manage  their  own  computer  facility? 

2.  Does  tne  company  use  terminals  which  tie  into  large  facilities  (not 
company  owned)? 

3.  Are  working  spaces  for  programmers  adequate? 

4.  Dues  the  company  maintain  separate  libraries  for  documents  and 
programs? 

5.  I.  c.ne  software  program  library  properly  equipped? 

6.  I;.  the  software  library  area  air  conditioned? 

7.  Are  Horary  storage  areas  overfilled  and  poorly  kept? 
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8.  Does  tne  company  maintain  a  ready  stoox  of  'fresh',  i.e.,  certified, 
magnetic  tapes,  paper,  eto.? 

Is  general  security  adequate? 

10.  Is  library  security  adequate? 

5.  Previous  Experience 

1.  Is  this  company  experienced  in  developing  Real  Times  Apd  .  lest:. or  s 
software  for  embedded  computers? 

? .  Has  this  company  had  experience  with  Government/Military  cont-a  tu? 

3  Has  the  company  developed  software  utilizing  MIL-S-52779? 

F\  Staffing 

1.  Is  the  Frogram  Management  staffed  with  a  Senior  System  Programming 
Engineer? 

2.  Are  the  lead  programmers  PHds  and/or  System  Engineers'7 

3.  Are  the  assistant  lead  programmers  staffed  hy  MA  degrees  :  -  Computer 
Science  and  related  fields  with  5  or  more  years  experience? 

3.  Are  the  algorithm  programmers  SA  degrees  witn  3  or  more  yea-s 
•exp  ' ~ier.ee  {or  equivalent)? 

9.  Are  tne  programming  language  spaoialists  equivalent  to  BA  degree?  with 
3  cr  more  years  experience? 

6.  Does  the  test  engineer  nave  a  BA  degree  and  5  or  more  years  exoerieoce? 

7.  Does  the  clerical  help  for  the  lead  programming  team  reflect  good 
stenographic  quality? 

8.  Are  the  librar ian/llbrary  personnel  qualified  librarians? 

9-  Are  Configuration  Management  Engineers  qualified  by  a  BA  degree  and  5 
or  more  years  experience  (or  equivalent)? 

10-  Are  the  3QA  engineers  experienced  in  QA  and  in  software? 

1).  Are  the  Specifications  Writers  MA  or  PhD  level  of  education  with  9  or 
move  years  experience? 

17.  Is  a  senior  Mr themalician  (PhD)  available  as  part  of  the  spool;  icatio" 
team" 

G.  Management 

1.  Does  the  company  present  a  written  Project  Management  (PM)  Plan  as  an 
example-7 

2.  Is  a  project  manager  designated  or  proposed  to  be  designated  for  this 
project? 

3.  Are  names  and  telephone  numbers  of  key  people  made  available  through 

the  PM  Plan? 

Is  &  clear  description  of  ’ worKflow’  presented  In  the  Project 
Management  Flan? 

9-  A'-a  project  management  review  procedures  specified  hy  the  FM  plan 
adequate? 

6.  Does  the  company  project  management  plan  indicate  that  an  established 
informal  review  process  is  working  io  this  company? 

7-  Are  formal  project  reviews  held  within  the  company  as  a  matter  of 

po  wicy  .* 

c  Are  or ooedu.-er  tor  company  reviews  clearly  soelled  cut  in  company 
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9.  I-'-  a  clear  WorK  Breakdown  Structure,  matrixed  to  Individual  SOW  tasks, 
presented': 

10.  Can  tne  Wortc  Breakdown  Structure  be  adequately  related  to  specific 
cost  factors? 

11.  Is  Work  load  Analysis  data  collected  aod  evaluated? 

12.  Dots  t da  management  nave  adequate  tools  for  monitoring  cost  trends? 

13.  Are  milestones  clearly  identified  in  the  project  managers  plan? 

14.  Is  ao  Assignment  Plan  Responsibility  Matrix  with  correct  time  phase 
relationship  available  and  presented? 

15-  It  tne  project  manager's  relationship  to  other  organizations  defined? 

16.  Is  the  end  product  deliverable  specifically  identified  in  the  project 
management  plan? 

H.  Subcontractor  Control 


1.  Does  tne  contractor  plan  to  subcontract  this  project  software? 

2.  Does  the  subcontractor  present  a  written  set  of  specifications  aod 
s  tandards  for  subcontracting  software? 

3.  Are  these  specifications  and  standards  in  compliance  with  MIL-S-52779A? 

I. _ Library  Change  Control 

1.  Does  the  company  have  a  written  set  of  Procedures  for  Library  Change 
Control? 

2.  Are  change  pages  on  each  document? 

3.  Is  a  current  catalog  maintained  of  all  project  software  documentation? 

4.  Does  this  catalog  provide  for  current  configuration  change  status? 

5.  Are  minimum  classified  security  requirements  met? 

6.  Does  tne  library  change  control  procedure  Clearly  lnter-relate  with  CM? 

7.  Is  a  stringent  control  of  computer  baseline  documentation  evident? 

8.  Is  a  'matcned  document'  receipt  control  procedure  provided  for  all 
changes? 

9.  Is  gooQ  traceability  apparent? 

J.  Library  Security 

1.  Are  library  security  measures  written  and  explained? 

2.  Are  all  libraries  designated  as  limited  access  areas? 

3.  Are  access  control  points  designated? 

4.  Are  library  custodians  named  and  designated? 

b.  Dc  library  custodians  nave  the  proper  level  of  security  clearance? 

6.  Is  a  library  authorized  access  list  maintained? 

To  a  i.og  maintained  of  all  access  to  tne  library  area? 

8.  Are  safes/safe  doors,  maintained? 

9.  Is  fire  control  adequate? 

K.  Document  Library  Facilities 

1.  Does  tne  company  have  a  separate  technical  library? 

2.  Will  a  specific  section  of  the  technical  library  be  dedicated  to  tnis 
project? 

3.  Are  storage  areas  and  shelves  adequate? 

4.  Is  lighting  adequate? 

5.  Are  security  safes  available  for  olassified  material? 

6.  Is  the  library  witnio  reasonable  proximity  to  the  users? 
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L.  Computer  Library  Facilities 

I.  Does  the  company  have  a  computer  program  library? 

2-  Does  the  compe.-y  have  adequate  storage  facilities  for  magnet 1 c  tape 
media,  etc,? 

3*  Are  ’safe"  areas  designated  for  master  tape  storage? 
u .  Are  fire  protection  facilities  adequate? 

5.  Are  computer  terminals  available  for  off-line  storage? 

6.  Are  'working  media'  kept  separate  and  distinct  from  baseline  tapes? 

7  -  Is  the  library  within  -easonable  proximity  to  the  working  ore;? 

A.  Are  ’scratch'  tapes  kept  separate  from  working  tapes  and  marte*-  taper  - 
Does  the  library  nave  good  magnetic  ‘ape  cleaning  end  repar 
favl  ties? 

Is  the  s* or -age  area  air  conditioned  with  humidity  control? 

II.  If  off-line  storage'  is  used,  are  the  .same  quality  factors  o-e?e»t 
/"tv  if  storage  facility? 

vp  Software  Quality  Control  Auditing 

Are  written  S2C  Informal  Auditing  procedures  presented? 

Are  3QC  personnel  penmitted  to  review  structured  walk-throughc 
1.  Are  regular  SQC  Audits  made? 

H.  Is  an  SIC  Aucit  Cer tiflcatioo  Report  Form  available? 

Is  '  JC  given  a  full  set  of  all  project  relevant  Policies,  plans, 

Pr  ". •.•tines  arc  procedures? 

i.  Does  SCC  nave  access  to  all  relevant  and  required  *'iles  arc  records? 

?,  Does  SOC  have  an  SOA  Audit  Stamp? 

•>.  Do  all  lies  contain  an  SQC  Audit  Record  showing  date  and  auditor? 

5,  if  a  formal  written  procedure  available  for  formal  SQC  Audits? 

.0.  Are  formal  minutes  of  audits  kept  in  SQC/ A  riles? 

11  Am  act' on  item's  properly  flagged,  and  tracked? 

IS,  Are  SQC  members  present  at  all  formal  Technical  Review*? 

1?.  Am  formal  age-das  published  in  advance? 

Does  the  routing  reflect  the  proper  organization  st-uoture? 

N.  Software  Q uallty  Assurance  Support  Tools 

I.  Are  "tools”  described  for  performing  "Operations  Research"  and 
rtems  Analysis"  ot'  the  performance  of  software  development  groups? 

2.  Are  techniques  described  which  augment  analysis  of  functional  and 
performance  requirements? 

3,  Are  "tools"  described  which  augment  analysis  of  functions'  and 
per  f  ;<"a«noe  r  equir  emer. t s? 

b  Are  software  optimization  tools  available  and  utilised? 

Are  tools  a-d  -'•oceovres  available  which  would  verify  spec  :.fi  cat  inn 
traceability? 

A,  Doe',  the  contracten  nave  a  "Coding  Conventions"  manual? 

7.  Are  any  or  all  of  the  above  documents  available  for  examination? 

8.  Are  toe  manuais  cited  that  are  available  for  examination  certified  by 
rn  S3 A  Stamp? 

P.  Is  clear  evidence  available  which  would  verify  that  the  to  Is  nave 
1  ,-«•  r  previos. sly  <Jeve  Vppp  ?■  re  •'.re  r,onms!.?.y  utilized? 
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-j.  A.  .  tool  otner  tnan  tnose  ment.onea  In  irt^s  reference  section? 

Arc  tne  auditiooa 1  tools  valid? 

■  2c  .r.e  ow A  support  tools  descrioed  adequately  meet  tne  requirements 

lor  SQA  support? 


I-  .i  wr.tten  cert il'ication  procedure  made  available  to  tne  evaluator? 
c.  D;  ■-  j  tn*a  procedure  reflect  good  SQA  Policy? 

3-  i;  tne  certification  procedure  valid? 

.  li  tne  car tifioation  done  by  independent  parties? 

Are  good  certification  standards  developed  for  eacn  project? 

6.  1:  a  Quality  Certification  Stamp  utilized? 

?.  Testing 

a  testing  policy  wnicn  is  written  anG  explained,  available  to  tne 

evaluator'. 

2.  -  -  ’Top  town'  testing  practiced  as  a  policy  by  tnis  company? 

test  plans  and  procedures  initiated  during  tne  specification  pnase 
s.  Ar.;  all  test  plans  and  procedures  routed  to  SQC  for  approval? 

р.  Are  tne  three  types  of  testing  properly  identified? 

с.  Art  provisions  for  continued  updating  of  test  plans  and  procedures 
mace? 

7.  Are  all  .eats  cone  by  independent  parties? 

5.  I.  test  result  feedback  reporting  adequate? 

?•  W.  .  logs  ano  records  be  maintained  by  test  personnel? 

.0.  Ltv.  testing  goals  sod  objectives  identified? 

...  Aro  goon  certification  testing  procedures  established? 

12.  Is  final  certification  tasting  to  be  done  by  an  independent  group? 

1.  report. ng  of  test  records,  test  reports,  and  test  results  as 
proposed  , >  tnis  company  adequate? 

Iw ..  *  .1  complete  system  tests  be  re-activated  after  each  trouble 
corredtio 


\  6  G  V,  v  v  A  0 1, 1 G  r, 


l  t-rc/GD^e  reports  be  Jti  i  izec? 

2.  V  . l  all  trouDves  found  be  duly  recorded  in  e  project  iogbooK? 

.1  trouble  reports  be  routed  to  6QC ,  fM,  and  otner  relevant  company 

off .car  s? 

A.  T  good  feedoacK  reporting  evideot  in  tn.s  descriptions  of  their 
pol.cy  fo:  .software  corrective  action? 

*  i  so.  .w, ire  designers  utilize  trouble  analysis  reports  to  'upgrade' 
tne  quail  ■  uf  .'.r.e  fine  i  product? 

Arc  trouble  analysis  reports  to  be  submitted  to  peer  appraisal  via  tne 
structure  walK-tnrougn? 

A'e  all  EGAs  listed  serially  on  developmental  listings  of  computer 
programs? 

o.  A.r --  trouo.es  to  be  reported  to  tne  Configuration  Nanagemont  Group? 

?.  A."  s  trouble  corrections  to  be  properly  reflected  in  ECHs,  ECPs  and 

EGAs? 
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R.  Product  Evaluation 

1.  Does  the  company  present  a  written  Product  Deliverable  Policy? 

2.  Does  the  company  perform  an  internal  functional  Configuration  Audit  of 
the  deliverable? 

3-  Does  the  company  perform  a  Physical  Configuration  Audit  o f  the 
deliverable? 

i.  Does  the  company  certify  each  program  tape  on  new  certified  media? 

?.  Are  fioal  printed  material  subcontracted  to  a  commercial  o-  intlng 

company? 

6-  Are  final  printed  materials  done  by  a  company  owned  printing  facility? 

7.  Are  complete  baseline  records  maintained  by  the  company  for  the  legal 
time  reouiremeot  (7  years)? 

3.  Are  final  prioted  material  presented  in  duality  bindings? 

9.  Are  final  printed  material  in  a  professional  bold  face  type  (not 
typewritten)? 

10.  Do  fioal  printed  materials  comply  with  the  standards  and 
specif : cations? 

11.  Do  all  final  deliverable  copies  provide  for  a  Quality  Cert. f icatioo 
Stamp'; 

'■2.  Does  the  company  guarantee  that  deliverables  made  on  this  project  will 
oea*.  tna  company's  highest  standards? 
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v.  GOT  ra-.L  i*  „■  oagemeot 

Sect .o  ,  ? .  Source  Selection  Evaluation  noarc 

7-1.  .-teview  AFR  70-15  Source  Selection  Policy  and  Procedures. 

7-2.  Prepare  s&ftware/nuraware  source  selection  criteria. 

'-G-  Participate  in  preparing  tne  computer  system  detail  standards. 

7-t.  Review  eacn  offeror's  proposal  from  a  software  viewpoint. 

7-5.  review  tne  CrOP  suomittea  »itn  tne  proposal. 

7-6.  A use os  eaon  proposal  against  tne  standards  and  criteria  previously 

estao . .  sne_. . 

7-7*  Fill  out  t  tie  evaluation  wo  rx  sbeets. 

7-8.*  Prepare  Item  and  Factor  summaries  as  appropriate. 

7-9.*  Prepare  Clarif ications/Deficienciea. 

7-10.*  Prepare  Points  for  Negotiations. 

7 - *  Eva.uate  response  to  Clar-ficatioos/Defieieoeies. 

7-12.*  Prepare  Final  Item  4  Factor  Summaries. 

7-ip.  Participate  in  negotiations. 

7-n.  Provide  deDriefing  inputs. 

7- -5*  Jrov  ..de  SSrlri  and  SSAC  report  inputs. 

7-16.  Assist  in  3SEB  briefings  to  SSAC  and  SSA. 

•N07S:  Sone  procurements  may  vary  from  tr.is  to  a  degree. 

I 
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Sect! on  8.  Computer  Program  Development  Plan 

8-1.  Review  CFDP  Data  Item  Description  (DI-S-30367A). 

3-2.  Review  other  program  CPDPs  for  appropriate  DID  modifications. 

8-?.  prepare  changes  to  the  CPDP  DID  to  be  provided  as  backup  to  the  CDPl. 
Tailor  to  t tIs  particular  program.  (See  attached  model  backup  sheet) 

8- a.  insure  the  CPDP  meets  requirements  in  toe  AFP  dOO-lK .  Vol  II,  Section 
3-9- 

3-5.  'nsure  the  CPDP  follows  the  guidelines  set  forth  in  ESD-7P-~7-263, 
.-oitva.va  Accuisition  Management  Guidebook,  Verification,  Para  P.3-  • 

■9-  .  Tnsur  that  the  CPDP  is  delivered  with  the  proposal  and  oadle  part  of  the 


sure  that  the  CPDP  is  put  in  the  SOW  and  in  the  CDRL. 

v  the  Computer  Program  Development  Plan  Checklist,  Figure 
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Fig  ,re  .  i-o--.  Coniputer  Program  Development  Plar  (.CPDP)  Guide  / Checklist. 

Defi  r.iuior, ;  The  Computer  Program  Development  Plan  (CPDP)  is  a  document  in 
wnich  the  .  .jntraotor  descr.oes  his  specific  detailed  plan  for  the  management 
and  development  of  ail  of  the  computer  programs  and  associated  documentation 
that  i.;  needec  .or  completion  of  the  contract;  this  may  include  support 
software  such  as  simulations  and  analysis  tools,  as  well  as  operational 
programs  and  automatic  test  equipment  programs.  (Source:  DID  DI-S-30567A) 

Genera . 

1.  If  the  contractor  uses  his  own  format,  is  there  a  cross  reference  between 
the  CPDP  and  the  items  outlined  in  the  DID  (DI-S-30567A)  Paragraph  10. i? 

2.  Does  t.ne  C?^?  snow  the  prime  contractor's  organizational  structure  to 
include  his  program  office  for  the  subject  contract? 

3.  Are  a*  of  toe  ouocontractor ' s  organizational,  responsibilities  and 
structural  relationships  to  tne  prime  contractor's  program  office  clearly 
ider.tif  iea  V 

*.  loss  f  •.!  contractor  provide  for  traceability  control  from  the 
A-Spec  if ic-ation  through  tne  C-Specification? 

5.  Is  one  oiAP  incorporated  into  the  CPDP?  (If  the  answer  is  yes,  use  the 
SQAP  checklist,  Figure  II-9-3,  in  conjunction  with  this  list.) 

6.  How  are  you  using  tne  CPDP  to  manage  the  development' effort  at  both  the 
contractor  and  Government  levels?  How  are  you  as  a  Government  activity 
tracking  tr.e  software  development  using  the  CPDP  provided  by  the  contractor? 

Are  any  of  the  management  tracking  functions  delegated  to  the  AFPRO  or  DCAS 

offices? 

7.  Is  tne  u?DP  current? 

o.  Are  Tig  meet  mgs  (Technical  Interchange  Meetings)  provided  for  in  the  CPDP? 

9.  nrs  th«.re  any  restrictions  in  rights  which  may  preclude  delivery  of 
software  maintenance  or  other  tools  to  the  Government? 

10.  Are  procedures  and  criteria  for  determining  software  development  progress 
described? 

11.  Do  tne  -wA  procedures  provide  for  the  review  of  software  documentation? 

12.  Are  system  integration  studies  to  be  performed? 

13*  uo  the  owA  procedures  assure  the  conduct  of  design  walk-throughs? 

1*4.  Does  tne  GQA  program  contain  procedures  to  identify  conflicting 
requirements ? 

15.  Does  tne  te3t  program  include  provisions  to  stress  test  computer  equipment 
and  compute^  program  configuration  items? 
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16.  Is  firmware  treated  as  software? 

17.  Does  the  plan  allow  for  unscheduled  audits? 

18.  Are  formal  qualification  test(s)  required  to  verify  CPCI  performance 
against  the  development  specifications? 

Requirements  Assessment  Summary 

19.  'Joes  this  section  snow  the  contractor's  understanding  and  assessme^*'  of 
the  completeness  and  clarity  of: 

=  Computer  resource  requirements 

b.  Definition  of  the  hardware  and  software  configuration  items  and  their 
interfaces 

c.  Current  functional  allocation 

Sizing  and  timing  analysis  in  order  to  meet  reserve  memory  requirement 

20.  Is  there  a  summary  of  high  risk  and  uncertainty  issues  (i.e.,  technical, 
cost,  schedule)? 

21.  Does  the  contractor  identify  proprietary  computer  resources  and  other 
used-as-is  or  modified-and-used  computer  resources  to  be  used  during  the 
contract  including  subcontracted  resources? 

82.  Are  there  approaches  with  backup  alternatives  to  minimize  or  overcome  tha 
risk  and  uncertainty  issues? 

23.  Is  the  r'isk  involving  long-lead  items  and  their  status  identified? 

2k.  Are  trade  and  optimization  studies  yet  to  be  conducted  and  their 
relationship  to  the  computer  resources  identified? 

25.  Is  the  rationale  for  the  selection  of  the  computer  programming  ianguage(s) 
if  not  specified  or  if  different  from  that  specified  identified? 

26.  Are  subcontract  items  and  who  will  be  performing  the  subcontract  work 
identified? 

Project  Objectives 

87.  Are  allocation  functional  requirements  for  each  computer  plan  clearly 
identified  for  each  CPCI.  Does  their  statement  for  <»ach  Computer  Program 
Configuration  Item  (CPCI),  per  MIL-STD-483  identify  the  planned  allocation  of 
<ts  functions  and  Interfaces? 

:>8.  For  each  CPCI  are  there  budget  goals  for  timing  memory,  interface,  and 
channel  capacity  (including  redefinition,  when  required,  based  on  the 
con crav tor's  approach!  and  margins  for  growth  in  these  capacities  V th  during 
c'*v  >ic,  nr>nt  and  future  operation? 
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..re  fa  ors  auon  as  reliability ,  maintainability ,  testability,  efficiency, 
portaoiiity  ,  interoperability,  and  other  factors  from  a  technical  approach 
discussed? 

nork  Befin.,-.  Ion 

3>).  -/oei  tie  CPDP  provide  for  a  feasible  work  breakdown  structure  for 
accor.piishir.g  the  development  effort  of  the  computer  program  development 
effort?  1  hccommena  a  3  level  breakdown  structure  for  tracking  software 
development . ) 

31.  toes  t.v  plan  identify  the  necessary  development  steps  such  as  analysis, 
design,  coting,  checkout,  integration,  test,  acceptance  delivery  for  each  CPCI 
ana  ma„or  supporting  computer  resources  and  their  relationship  to  the 
contractor's  Work  Breakdown  Structure  (WBS)  per  MIL-S7D-881,  including 
required  Woo  deviations? 

32.  !.re  Uo-.s  associated  with  support  functions  (such  as  documentation, 
configuration  management,  data  management,  management  reviews,  software 
q^ai.ty  assurance),  anc  their  relationship  to  other  contractual  tasks 

identified': 

33.  Are  tan.-.s  involving  integration  and  test  of  CPCIs,  including  such  factors 
as  separate  test  configurations,  system  tests,  and  operational  tests  where 
applicable  identified? 

3h.  Are  major  development  steps  for  all  identified  Computer  Program  Components 
(CPCs)  in  each  CPCI,  and  all  elements  of  the  required  computer  resources  which 
support  the  product  identified? 

Worx  ocnecU  . e 

3b-  ice.,  tr.c  plan  provide  a  time  schedule  of  the  work  elements,  based  or,  the 
contract  ra.-ater  schedule,  indicating  initiation,  intermediate  (e.g., 
a/ai  ,.u.,i..i:  •  of  draft  and  final  copies  for  formal  and  informal  documentation) 
ana  .ample*  .on  t-mes  for  ail  CPCIs  and  time/performance  critical  Computer 
Program  Components  (CPCs)  including  formal  and  informal  milestones,  reviews, 
audits,  Key  meetings,  and  documentation  releases? 

Activity  NeiworK 

36.  boos  t rc  contractor  provide  an  Activity  Network  (e.g.,  PERT)  compatible 
with  tne  Wc r,<  Scneduie  of  computer  program  development  efforts? 

37.  are  interface  activities  (suen  as  hardware  development)  addressed  as  part 
of  the  Activity  Network? 

33.  Are  critical  path3  or  near  critical  path  elements  identified? 

Organizdtio n 

39.  Coes  tre  contractor  provide  for  delineation  of  the  contractor's 
organizational  structures,  authorities,  responsibilities,  interfaces,  skill 
requ. remenis  ana  ime3  of  communications  necessary  to  manage  and  execute  the 
scheduled  activities  to  assure  proper  task  completion? 
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the  relationships  between  the  contractor  and  independent  or  -edun d  ,n 
'"-t:  Mention  ind  validation  groups,  other  contractors,  subcontract .  .--s  '  rv 

P  ’  •  ‘  r .  i .  .g  agency.  -1.  •  s.ip  port  agency,  anc  the  u-.er  .  gency  lerti'".  ■ 

?*»y  li«d  -s.sns.ger s  and  employees  identified  by  name! 

Fe  ■  ■“.  Allocation 

' -  "oes  the  cc  -.tractor  provide  for  allocation  of  facilities,  lptr-rotc"  >r 
tipe,  te«'t  equipment.-;,  and  other  relevant  resources  a  grin." 
cr  r ■ ”  t  -i  streotgras,  wort  elements  and  schedules? 

U>.  Pee1'  th«  contractor  identify  projeot-pec.u.’.iar  resources  require  ,  sue h 
u _  purpose  hardware  and  computer  programs,  Gc  v-'mmer.t-  f  uri ‘.  she  1*1  nr. 

‘  .  "  '  f»  p  ? 

'Joes  the  contractor  identify  items  which  may  impact  resources,  rjor.  an 
•  '•'.'•!<  de/tlopxert  items,  special  security  requirements ,  subcontract-**’ 

■V.-.  ?.*  there  a  presentation  cf  the  methods  to  be  used  to  assure  eonratisi)  t 
cf  ..c  pr-cc  ured  system?  with  the  intended  operational  physical  f»e_  iti-  s? 

r j peering  f-tandards 

Ml.  ices  the  contractor  provide  a  definition  or  identification  of  software 
engineering  practices  as  they  nooly  to  each  CFCI  and  computer  resource  •"..•ecu 

iV.t:  the  contractor  identify  how  these  practices  will  be  maintained  or  r. 
*  vi’.l  :<*  ??3{.*?’3d7 

i 'pstpl es  of  engineering  practices;  standards,  conventions,  procedures,  and 
rule r;  for  program  design,  structures,  display  and  logic  standards, 

Input /o .ltput  standards,  guideline?  for  program  sub-uivlsi or,  coding 
techniques,  and  programming  languages,  date  base  standard-  anc  othe- 
dc.  ciplines  affecting  development) 

l*es  1  £, :c  As su r- -.nee  Technique s 

tv.-es  the  contractor  provide  a  definition  of  the  techniques  user  for  desi 
am  lysis  arid  control  to  assure  completeness,  validity  traceability  ■«' 
ric.ilr  -rert'  ter ts.ci.lity  and  compliance  with  standards  a-’c  practi*.  -•? 

•i  o .  ”■•;’>  s  the  contr-actor(s)  explain  ris  app-o-o  *  lo  preliminary  and  critic:'1 

:'-s  '  ;  7.  r  ?  vi CVS'7 

o' a  the  conlrac'  c*-  define  what  he  means  rv  design  evaluation  " -chain uev 
;•••••  .n  lent-  fication  of  the  purpose,  appl* ov„ion ,  a<»«j  validity  of  .  be  to -“•Is 
used  ; j..a.,  computer  resource  support  products,  simulation. ,,  prototype 
•  ;■  >n  -tica.l  models,  and  osrulations)  ? 

c- s  the  c -t  •  -  ..nov  identification  cf  verification  and  va'  •  bailor 
s?  ..-.d  ai’i”  one  ro  -  they  will  be  qua),  if  led  and  used.  •  taring  .re  » it  i  r 

.  *.4".  1  •hAOqg'/ 
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dj.  .3  .ne.r  a  abatement,  of  the  plan  for  defining  managing,  controlling ,  a.nc 
reporting  i  no  status  of  budgets  on  timing  and  memory,  and  technical  interfaces 
and  interface  margins?  Are  guidelines  and  checkpoints  for  insuring  future 
computer  program  growth  oapuoiiity ,  nodularity,  and  ease  of  modification 
adequately  tier.tir  ieu? 

Petal -eu  br-.ign,  1’oalng,  and  Checkout 

be*  Doe  i  tr  contractor  provide  a  definition  of  tne  procedures,  steps  and 
aocua;en  .3  associated  witn  detailed  design,  coding  checkout,  review  and 
acceptance  f  the  individual  modules  comprising  CPCs  and  CPCIs  including  the 
functions  /  ■’formed,  the  handling  of  the  internal  and  external  interfaces,  the 
control  of  'arameters?  (i.e.  ,  memory  allocations,  execution  time  allocations, 
channel  retirement  allocations,  operating  sequences,  data  base  requirements 
and  test  rec ui resents) . 

5u.  bu ej  tr:  contractor  identify  computer  programs,  equipment  and  devices 
requ-red  tc  support  system  computer  hardware  ana  to  facilitate  software 
oranges,  iriiuoing  diagnostic  software  for  all  computer  resources? 

55.  Does  tr e  contractor  identify  criteria  and  the  mechanism  for  acceptance  of 
moGules  for  integration  and  te3t? 

Development  Test  ana  Evaluation 


5b.  Does  tne  contractor  provide  a  presentation  of  the  integration  and  test 
philosophy  and  approach  for  CPCs  and  CPCIs? 

57.  uoe >  the  contractor  explain  how  this  philosophy  is  applied  in  the  design 
and  scheduling,  and  now  the  approach  leads  to  the  Preliminary  and  Formal 
Qualification  Tests? 

58.  Poes  tr.c  contractor  provide  a  definition  of  the  interfaces  anc 
responsibii  ties  (such  as  training)  among  test  groups  and  other  project 

part.cipantuV 


by.  woes  t; .  contractor  show  tne  resource  and  schedule  impact  of  independent 
verification  and  validation  activities? 


System  Teen  and  evaluation 


60.  woes  tr..-.  contractor  explain  procedures  for  the  coordination  and  support  of 
CPC I...  and  l  tier  computer  resources  during  formal  system  testing  (and  during 
Operations.  Test  and  Evaluation  as  applicable)? 

bl.  uoes  ti"  :  contractor  include  the  necessary  system  training  and  transfer  of 
>i  it  a  as  par  -  of  tne  system  test  and  evaluation  information? 


Anomaly  Cof"roi 

fey.  boi!j  t‘  ;  contractor  explain  the  methods  or  system  by  whlon  computer 
software  aroma lie 3  are  detected  and  documented  during  all  test  anc  evaluation? 

op.  does  t.  contrator  give  the  methodologies  of  error  correction  and 
configuration  control?  Does  he  identify  who  is  responsible  for  such 
methodology  r3? 


62 


HELP.?,  Volume  II  1  ~P  ? 

Contract  Management 


fc*i» .  Does  the  contractor  explain  the  data  to  be  gathered  to  enable  assessment 
of  the  reliability  of  the  computer  resources,  retorting  period",  ri- ■:  •  s  .’-yes, 
And  method  to  establishing  when  each  anomaly  (i.e. ,  design,  ceding,  tec-  ., 
etc.;  was  detected  and  corrected? 

Me  r ■  > gem- ? n t_  C ontrols 

6p.  Do  i  the  contractor  explain  the  relationships  between  the  manage  ?ant 
i.x-'.t e  of  tne  CP  DP  and  other  applicable  management  plans'’  (i.«.  • 
lore'.'.",  cation  Management,  Security,  and  Systems  Engineering  Manage^  ">«•  unr. 

"/  > . 

ci,  ;>>cs  the  contractor  explain  the  means  and  criteria  for  manage^er t 
.*  -.a»':.s.ti?nt  end  cortrol  of  the  development  croc  c.c3'>  (Including  subcontract-..--'" 
mechanisms  ?or  initiating  management  actions  when  status  d.cfte: 
d  .ten  from  plan,  such  as  resource  reallocation ,  schedule  si'1  no*,  p-?,  c  r 
»s.  or  performance  degradation) . 


Docv mentation 

ft?  Pop?  the  contractor  provide  any  new  software  documentation  tools  or  a 
description  of  the  contractor  documentation  practices  including  techniques 
:  >-.t  "de<:  for  use  on  the  subject  contract? 

68.  Does  the  contractor  identify  the  specific  documentation  alternatives  and 
the  .necessary  changes  to  the  content  and  format  of  the  specified  a>'.ur‘-r.tation 
requirements?  (Include  as  a  minimum  a  description  of  the  equivalent/  :• -mures 
between  the  specified  documentation  requirements  an-:  the  proposed 
a  . .  .""rati  vc(  s } ) . 

6;.  'iot'.j  the  contractor  provide  a  description  oh  the  procedures  w’-l-th  v;  J. 
used  'c  keep  the  informal  documentation  current? 

7C.  Dees  the  contractor  provide  a  description  of  the  approach  tc  sc-pvtc” 
program  in-line  commenting? 

Conf iguratlon  Management 

71,  Does  the  contractor  provide  for  the  special  aspects  of  computer  software 
configuration  management  net  addressed  in  the  overall  Configuration  Management 
f  (Include  tnc  organizational  placement,  change  deviation,  wa? ver 
authorization  and  control,  internal  engineering  orange  board,  conf l/ u ratio.! 

‘  C(.  -tif  ..c-ati-rn,  computer  program  lib-ary  control.  t  ue  cor'  -oj.  or-  ng-*’  fl¬ 
an,  jor. -del i  .•eraole  computer  program-,  the  ’-""ulinc  c'  i->t-'- face-  c-  v  . 
between  the  rare, ware  cud  the  software  of  the  system,  maintenance  C 
ind-.  -uerdent  master  card/d-ccks/tapes/disks,  ocriro  of  development  vr-o  •  -nr  o' 
Cr'-Cs  <rd  CvCj  ,  handling  of  source  ant.  object  control  over  test 

r-  \t  ro-ments,  relationship  to  programming  ftardardr.  and  -e’a.  lor-s‘-  v  t  - 
r.  -nua.  ity  assurance  functions,. 
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Vena. >r/  'r'r.  computer  riesources 


'Id.  boos  the  plan  state  the  oror-ebures  for  qualifying  and  documenting  the 
present  adequacy  and  the  adequacy  for  future  use  of  vendor-supplied  or  GrE 
computer  resources  ( careware  and  software)  and  the  means,  such  as  testing,  for 
accomodating  revision  of  vendor-supplied  computer  resources? 


port  Serources  for  the  Deployment  Phase 


73-  Does  th«  plan  provide  for  the  recommended  support  philosophy  and  the 
resource  requirements  for  use  after  the  FEED  phase?  Does  this  section  of  the 
CPDP  summarize  or  specifically  reference  the  plan  for  the  transfer  of  computer 
resources  including  support  software  and  tools  to  the  appropriate  Air  Force 
agencies? 
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Figure  II-8-2.  Computer  Program  Development  Plan  Backup  3h eetr 

BACKUP  SHEETS 

Data  Item  DI-S-J0567A 
Computer  Program  Development  Plan 

The  contractor  shall  substitute  the  following  for  3ox  10  of  the  DID: 

Detailed  Requirements:  The  contractor  shall  provide  the  following  information 
in  the  format  specified  'oeicw: 

TABLE  CP  CONTENTS 


Ci'CSS  REFERENCE  TABLES:  The  tables  shall  trace  the  delivered  CPD°  to  th«  data. 
iv3tr.  requirements  defined  in  System  -Specification  XXX  and  the  Statement  o' 

V<-i (SOW). 

.1,0  ORGANIZATIONS,  SCHEDULES,  AND  RESOURCES 


A.  ORGANIZATIONS 

(1)  Show  the  prime  contractor’s  organizational  structure  to  include 
his  program  office  for  the  subject  contract. 

(2)  Show  all  subcontractor’s  organizational  responsibilities  and 
structural  relationship  to  the  prime  contractor'3  program  office 

(3)  Show  the  structure  and  relationships  within  the  software 
development  organization.  Describe  the  following  are:::': 

(a;  The  specific  responsibilities  of  each  element  of  the 
software  development  organization. 

(b)  The  number  and  allocation  of  personnel  involved  in  toe 
design,  coding,  testing,  documentation,  quality  control  and 
maintenance  of  all  project  software.  The  description  shall 
include  joo  titles,  job  descriptions,  Job  responsibilities. 
3kills  required,  and  the  relationships  of  each  ..  o  position 
to  others  in  the  software  development  oeganiratK  n. 

(c)  The  procedures  for  notification  to  the  °rincipal 
Contracting  Officer  (POO)  that  will  be  applied  concerning 

'  •-v.sc'l  or  r-3\iov<*d  e  r'orr*  Vif»  -'o;-  nr' 

o  r  ?  r  ?.  z  a  *:  1  c  r.  ?.  \  s  t  c  Vi  re . 

(d)  How  erganizat lonal  independence  of  Software  Qua1.'  ty 
Assurance  •  3QA )  personnel  will  b"1  achieved. 
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8 .  Sv-hEDULES 

Proviso  a  detailed  sc beau *e  for  tne  availability  o:’  computer  hardware  ana 
:'or  the  development  of  software.  ’Phis  schedule  will  be  a  graphic 
representation  oz'  tne  aetaii  development  milestones,  scheduled  activities  and 
oeliverab^t  ,-..  Ir.aicate  on  the  scnedule  the  time  allocated  for  the 
Government  s  review  of  software  related  data  submissions.  The  schedule 
shaiiincluf  a  the  time  allocated  for  each  phase  of  computer  program  development 
(e.g. ,  ana. /sis,  design,  coding  and  checkout ,  etc.)  oy  1PCI.  If  Government 
Furnished  equipment  (GFE)  i3  to  be  provided,  indicate  on  the  scheuule  when  the 
GFE  will  Dc  requirec. 


C.  DLLIVERAoLSS/DGCUMSNTATION 

(1/  (a)  Propose  a  schedule  for  developing  computer  program 

Documentation  m  preliminary ,  interim  and  final  versions. 
Include  in  tne  schedule  Government  reviews  and  Technical 
Interchange  Meetings  (TIMs)  to  aid  m  transitioning  from 
the  preliminary  documentation  version. 

(b)  Describe  a  procedure  to  be  used  during  the  review  period  to 
resolve  the  Government’s  technical  concerns. 

(J/  Describe  tne  software  documentation  approach  which  will  be 
followed  to  meet  or  exceed  the  requirements  of  MIL-STD-463 
(USAF)  for  documentation  of  85  and  C5  specifications.  If  new 
software  documentation  tools  or  techniques  are  to  be  employed, 
provide  a  description  of  these  tools  and  techniques  and  explain 
any  restriction  in  rights  which  may  preclude  delivery  of  these 
tools  to  the  Government.  Discuss  how  this  documentation 
approach  will  be  consistent  with  the  development  approach  in 
providing  effective  documentation  of  the  software. 

i;  Describe  the  •.pproacn  to  computer  m-line  commenting*  including 
the  time  at  wr.icn  it  is  initially  accomplished  and  the  plan  for 
maintenance,  include  tecnmques  for  identifying  changes 
included  in  each  re-compilation/re-assembly  of  each  CPCI. 

(‘j  Descrioe  the  procedure  for  mapping  of  functional  requirements  to 
GPCIs . 

MGTt;  Insz  re  approac.n  is  consistent  with  Paragraph  3*3-3  of  System 
Specification  in  MIL-STD-483,  Appendix  VIII. 

D.  PROGRESS  MONITORING  AND  STATUS  REPORTING 

(1,  De3crioe  the  procecures  and  criteria  for  determining  software 
development  progress. 

(f.  Describe  your  approach  to  software  development  progress 
reporting. 

Describe  your  approach  to  identifying,  reporting  and  resolving 
software  development  problems. 
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(^)  Describe  the  methods  and  procedures  for  estimating  collecting, 
analyzing,  monitoring  and  reporting  the  sizing  and  timing  of 
CFCIs. 

(5)  Describe  the  conduct  of  all  internal  project  reviews. 

E.  CONTRACTOR'S  FACILITIES  AND  HARDWARE 

(1)  Describe  the  software  development  and  maintenance  facilities  to 
be  used  in  the  development,  testing,  and  maintenance  t  <=> 
computer  orograms. 

(2)  Identify  all  computer  configurations  including  peripherals. 

State  associated  delivery  dates. 

(3)  Identify  all  testing  resources  to  be  used  to  support  compute-- 
program  development  and  testing. 

( 4 )  Identify  all  software  tools  that  are  planned  for  use  .r  f~e 
development  of  the  computer  programs.  Indicate  whether  toois(.s) 
are  deliverable  or  non-deliverable  and,  provide  a  rationale  why 
the  development  tool(s)  are  not  deliverable,  if  applicable. 

List  any  restrictions  in  data  rights  for  the  software  tools. 

2.0  SOFTWARE  DEVELOPMENT  AND  TESTING  METHODOLOGY 

For  each  of  the  software  development  and  testing  methodologies  discussed 
below,  the  contractor  shall  identify  whether  he  has  past  experience  :n  'he  use 
of  this  methodology.  The  contractor  shall  distinguish  between  existing 
capabilities  and  those  capabilities  not  yet  developed  which  will  be  used  or 
Vi?-  subject  contract. 

A.  GENERAL  REQUIREMENTS 

(1)  Identify  the  proposed  programming  language  and  provide  a 
rationale  for  it3  choice.  If  mere  than  one  language  is  proposed 
(e.g.,  assembly  language  is  proposed  for  some  subsegments), 
provide  a  rationale  for  the  use  of  each  additional  language. 

(2)  Describe  your  approach  to  analyzing  system  requirements  and 
deriving  the  detailed  functional  requirements. 

(3)  Describe  any  teehniques/tools  for  allocatin',  of  requi*- en-<-  its  h-> 
hardware /software . 

(t)  Describe  your  approach  ''or  DOD  security  level  control'-  and 

methods  of  implementing  and  maintaining  ADP  security  plus  acv 
unique  security  problems  or  planned  installation  security 
requirements. 
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b.  DESIGN 

(i)  Insure  the  ush  rf  top-down  design  methodology  to  be  used  :n  tne 
design  end  implementation  of  the  system.  Describe  any  effect 
that  the  choice  of  language(s)  will  have  on  the  proposed 
methodology. 

(2;  Describe  how  the  proposed  computer  programming  language  conforms 
to  the  proposed  methodology.  If  an  automated  translation  tool 
will  be  employed  to  achieve  this  conformity,  describe  the  tool 
and  my  data  rights  restrictions  which  may  apply. 

Hj  Describe  your  approach  to  data  base  design  and  implementation. 
Also  describe  how  data  base  compatibility  will  be  maintained 
between  HOLs  and  assembly  language  programs,  if  both  are  used. 

(e)  Describe  your  approach  to  conducting  Preliminary  Design  Reviews 
(PDfis).  If  incremental  PDRs  are  proposed,  describe  how  they 
will  be  accomplished. 

(bJ  Describe  your  approach  to  conducting  Critical  Design  Reviews 
(CDRs).  If  incremental  CDRs  are  proposed,  describe  how  they 
will  be  accomplished. 

(0)  Descnoe  your  plan  for  ensuring  modularity  and  ease  of 

modification.  Also  include  a  description  of  the  plan  for 
controlling  computer  program  growth,  to  keep  it  within  any 
imposed  limits. 

C.  CODS  AND  DEBUG 

(*.'  Describe  your  approach  to  coding  and  debugging  the  software. 
Describe  any  progress  monitoring/reviews. 

( 1 >  Describe  any  software  development  libraries  to  be  employea. 

D.  DEVELOPMENT  TESTING 

(i)  Describe  your  approach  to  unit  level  code  testing. 

Describe  the  extent  of  in-plant  testing  including  test  planning, 
methods,  documentation  and  reporting.  The  general  procedures 
for  reporting,  monitoring  and  re solving  program  errors  and 
deficiencies  discovered  during  in-plant  test  should  also  be 
described . 

(;->  Describe  the  proposed  method  for  insuring  computer  program  test 
and  evaluation  (CPTAE)  testing  and  verification. 
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3.0  SOFTWARE  MANAGEMENT  AND  CONTROL 

A.  DESIGN  AND  DEVELOPMENT  CONTROLS 

Describe  your  Programming  Support  Library  (P3L). 

B.  CONFIGURATION  MANAGEMENT 

(1)  Describe  facilities  and  procedures  to  be  used  to  maintain 
configuration  control,  per  MIL-STD-483  (USAF),  during  the 
software  development  to  include  error  correction  control  during 
integration  and  test,  as  well  as  development. 

(2)  Describe  facilities  and  procedures  for  configuration  management 
to  be  used  after  first  system  acceptance,  if  applicable,  per 
MIL-STD-483  (USAF). 

C.  MAINTENANCE 

(1)  Describe  methods  for  in-plant  and  on-site  maintenance  of 
software  and  documentation. 

(2)  Describe  the  maintenance  approach  for  all  vendor  supplied 
off-the-shelf  software  (operating  systems  and  utility  services). 

D.  SOFTWARE  QUALITY  ASSURANCE 

Describe  your  Software  Quality  Assurance  (SQA)  Program. 

USE  SQAP  CHECKLIST  II-9-3. 
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Section  9  -  Software  Quality  Assurance 

i-I.  Review  MIL-S-5S779A ,  Software  Quality  Assurance  Program  Requirements. 

i-k.  Review  MIL-HDtiK-334,  Evaluation  of  a  Contractor's  Software  Quality 
Assurance  Program. 


1-3.  Review  MXL-STD-lo79  (NAVY), 
MIL-STD-1679  is  used,  care  must  be 
requirements  and  MIL-Specs/STbs. ) 


Weapon  System  Software  Development.  (If 
taken  to  avoid  conflict  with  other  contra 


1-4.  Review  the  Memorandum  of  Agreement/Letter  of  Delegation  Checklist, 
Figure  II-9-1. 


I- b.  Review  tne  Software  Quality  Assurance  Self-Inspection  Checklist, 

II- 9-<r. 

1-b.  Review  the  SQA  P.an  Che-klist,  Figure  II-9-3. 

1-7.  Review  ESDP  79-1,  Guidance  for  Applying  MIL-S-52779A. 


Figur 
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(NOTE:  This  Strawman  is  not  an  AFMCD  coordinated  nor  staffed  document.,' 

Figure  iJ-9-1 .  Memorandum  of  Agreement /Letter  of  Delegation 
Checklist  (MOA/LuD) 

Below  in  a  list  of  possible  items  f  ir  incorporation  into  an  M'A  or  ’ 

Tnia  list  should  :e  tailored  to  individual  con*. rant /program  needs,  od 
individual  CAO  capabilities.  For  example,  programs  with  'r:ti~ai,  high-rise 
S jf tware/f irmware  development  would  certainly  require  more  d.-siv » 
monitoring  than  those  with  validated  of  f-tne-shel  f  .  •*,'tw.ar«.  The  r---  "jpe  •' 
proposal  package  and  referenced  appl  tot.;. If  document:  should  in-  reviewed  nr  i  or 
to  selecting  items  from  the  list.  Obi i. gat  ion  of  the  contract y  to  correct 
deficiencies  as  a  result  of  CAO  monitoring  activities  will  varv  depending 
contractual  requirements. 

Any  resulting  list  of  items  should  pe  negotiated  between  the  program 
office  and  the  CAO  to  assure  sufficiency  of  monitoring  activities  and 
availability  of  CAO  resources  to  perform  the  functions. 

a.  If  provisions  of  MIL-S-52779A  are  on  contract  (applicable  paragraph  f-om  ! 
MIL-i>-52779A ) 

(1)  Use  Mlu-HDbK-diU  as  guidance  to  monitor  for  compliance  with 
MIL-S-52779A  requirements.  (Para  1.1) 

(2)  Review  SQA  Plan  and  Procedure;  for  compliance  with  Mi  --S-1  d 77 •)/,  (as  I 
tailored,  if  applicable).  (Para  3.1) 

(3)  Assess  the  contractor's  organization  and  review  position  descriptions 
to  assure  personnel  have  authority  and  responsibility  to  perform  SQA 
functions. (  Para  3-1) 

(4)  Audit  the  SQA  organization  to  verify  QA  audit  reports  are  being 
written  and  presented  to  management.  (Para  3.d.t>) 

(0)  Review  tools,  techniques,  and  methodologies  (TTM)  to  assure  they  are 
being  used  as  intended  (described  in  the  SQA  Plan  anu  Computer  Program 
Development  Plan).  Verify  TLMs  are  validated,  effective,  ami  accurate.  (Para 
:  .M) 

(f)  Monitor  for  compliance  with  procedures  on  use  of  tools,  teohni ones 
and  methodologies.  Audit  records  and  results  of  TTM  to  assure  feed  ha  b:  to 
contractor  QA  functions.  (Paras  3.2.1  and  3.2.6) 

(7)  Review  programming  standards  and  convention-,  to  assure  comp! < at oe 
with  contractual  requirements  and  internal  procedures.  Audit  coding  for 
compliance  with  programming  standards  and  conventions.  Audit  tailoring  of 
these  standards  and  conventions  for  individual  progr  ir„::  to  assure  1  low-down-  of 
reqii.  rements  and  authority  for  changes.  (Para  i.p.  U  . 

vb)  Audit  design  documentation  for  proper  reviews,  including  conduct  of 
internal  design  reviews,  prior  to  submittal  to  the  -lov-rnment,.  Audi:,  f 
:  ndei-eu  lent  internal  reviews  of  design  documents.  (p.ira  J...  ..  1 
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'•Hi  audit  for  traceability  between  software  requirements  and  software 
iesign  documents.  (Para  3.2. 2) 

llu)  Monitor  for  progress  on  tasks  as  described  by  project  schedules;  key 
a.  software  development  scnedule.  Keport  variances  as  they  happen  but  no  less 
frequently  tnan  monthly.  {Par a  3.2.3) 

(1~)  heview  design  documents  for  compliance  with  Contract  Data 
requirements  -1st  (CDHi.)  and  Data  Item  Description  (DID)  format.  Assure 
concurrent  development  of  contract  data/documentation  with  development  of 
software.  (Para  b .2.2) 

(,1c)  Monitor  existence  and  compliance  with  conf iguration  control 
procedures  for  design  documentation.  (Para  3.2.7) 

(13)  Audit  design  documentation,  data  dictionary  (if  available),  and 
source  listings  to  verify  that  naming  conventions  (as  described  in  the 
Computer  Program  Development  Plan  (CPDP)  and  internal  procedures)  are  being 
followed.  (Paras  3.2.2  and  3*2. A) 

(1A)  Assure  design  requirements  as  delineated  in  the  Statement  of  Work 
(SOW)  arid  A-Spec  (usually  Paragraph  3*3.8)  are  being  followed  e.g.,  top  down, 
modular,  commenting  conventions.  (Para  1.3) 

(15)  Monitor  tne  configuration  management  of  patches  and  assure 
replacement  (incorporation)  prior  to  delivery.  Assure  the  contractor  is  aware 
of  the  ground  rules  for  patches.  (Para  3-2.9*d) 

(Id)  Monitor  for  compliance  with  program  support  library  procedures 
(internal  and  tnose  In  CPDP).  Assure  versions  are  safeguarded,  identified, 
and  accurate.  (Para  3*2.5) 

(17)  Audit  for  contractor  SQA  participation  in  informal  and  formal  reviews 
and  audits.  Verify  follow  up  action  taken  on  items  resulting  from  reviews  and 
audits.  (Para  3-2.6) 

(Id)  Monitor  the  contractors  compliance  with  the  Configuration  Management 
Plan  and  CM  control  procedures  for  software  products.  (Para  3.2.7) 

(19)  Heview  Configuration  Management  Audits  and  determine  if  corrective 
action  is  identified,  adequate,  and  implemented.  (Para  3-2.7) 

C'D)  Assure  test  plans  and  procedures  are  internally  reviewed  prior  to 
submittal  to  Government.  Heview  for  compliance  with  DID  format.  Para  3.2.8.b) 

(21)  Witness/monitor  all  formal  software  tests  (PQTs,  FQTs),  integration 
and  systems  testing.  Sample  Informal  testing  to  assess  progress  and 
compliance  with  internal  procedures.  (Para  3.2.8) 

(2.)  Keview  test  reports  to  certify  accuracy  of  formal  test  results. 

(Para  i. 2.8. f) 
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123)  Audit  software  trouble  reports  to  assure  coordination  with  proper 
organizations,  ranked  according  to  severity,  adequately  described,  and 
properly  resoluted.  Review  documentation  of  "fix",  if  applicable,  as  well  as, 
results  of  retesting.  Examine  extent  of  retesting  to  determine  adequacy. 

( Para  j. 2. 9) 

124)  Identify  recurring  problems  and  rates  of  open/closed  software  trouble 
reports  and  Identify  trends.  (Paras  3-2. 9-c  and  3.2.9.d) 

(25)  Review  contractor  retest  decisions  to  verify  that  all  affected  code 
was  evaluated/retested.  (Para  3*2. 9. e) 

(2b)  Audit  deliverable  code  for  matcning,  verified  against,  tne  master. 
(Para  3*2.7) 

(27)  Monitor  tne  contractor's  method  for  identifying  critica. 
software /f irmware  for  subcontractors.  (Para  3-3.) 

(28)  Audit  tailoring  and  flow-down  of  software/f irmware  requirements  to 
subcontractors.  (Paras  3*3  and  6.2. l.c) 

(29)  Monitor  contractor’s  system  for  assuring  subcontractor 

organlzat lon/personnel  are  qualified  for  performance  of  SQA  functions.  (Para 
3.3) 

(30)  Audit  the  contractor's  system  for  compliance  with  the  GOA  Plan. 

(Para  3.1) 

(31)  Audit  contractor's  compliance  with  internal  computer  software 
policies  and  procedures.  (Para  3-D 

(32)  Review  contractor's  selection  of  support  software  and  computer 
hardware  to  assure  Goveernment  acceptability  is  established  prior  to  their 

use.  (Para  3-2.8) 


(1)  Audit  software  trouble  reports  to  assure  they  are  prioritized  by 
severity  as  required  by  Para  5-8. 5.2  and  categorized  properly  in  accordance 

with  5.8.5. 1. 

(2)  Audit  modules  for  completion  of  a  code  walk  through  prior  to  module 

testing.  (5.8.1) 

(3)  Witness/monitor  critical  module  tests  (identify  in  the  MOA/LOD). 

(5.8.1) 

(4)  Witness/roonitor  critical  subprogram  tests  (identify  in  the  Mi.'A  'l.O(') . 

(5.8.2) 

(5)  Witness/monitor  all  program  performance  tests.  (5.8.3) 

(6)  Witness/monitor  ail  systems  integration  tests.  (5.8.4) 
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(7)  Audit  tne  contractor  QA  organization  for  participation  in  design 
reviews  and  design  waik-tnroughs  and  following  up  on  action  items.  (5.9.1. 3) 

(8)  Audit  testing  to  assure  tnat  the  software  used  is  tne  current 
approved  version.  (5.10.2.2) 

(9)  audit  test  program  operation  (5.10.2.4),  duration  (5.10.2.5),  input 
luta  (5..0.CW  and  stress  (5.10.2.7)  for  compliance. 

(10)  Audit  testing  to  assure  conduct  in  the  proper  testing  environment  (in 
accordance  witn  Paragraphs  5.10.2.1,  5.10.2.8  and  5.10.2.9). 

(11)  Audit  retesting  for  conduct  in  accordance  witn  Paragrapns  5.10.2.10 
and  5.10.3* 


(12)  Audit  error  (Para  5.10.3.1)  and  patch  limits  (Para  5.10.3.2)  to 
assure  tney  are  within  requirements. 

c.  Miscellaneous: 


(1)  Use  ESD-TR-82-145  software  quality  metrics  handbook,  to  provide  an  | 
assessment  of  tne  software  for  critical  CPCIs  (PO/CAS  must  identify). 

(2)  Use  ESD-TR-80-1 15 ,  Vol  I,  II  to  help  monitor  the  sizing  and  timing  | 
analysis  reports.  Flag  borderline  and  discrepancies  in  meeting  contractual 
requirements,  significant  growth  since  last  analysis  should  be  reported. 

Audit  for  reserve  growtn  capability. 

(3)  Audit  software  development  folders  (for  compliance  with  CPDP)  to 
assure  proper  contents  and  updating  of  material. 

(4)  Monitor  recording  and  reporting  of  work  breakdown  system  cost  data 
and  audit  the  data  for  accuracy. 

(5)  Monitor  cnanges  (growtn)  in  number  of  lines  of  executable  code 
compared  to  estimates  at  time  of  CDR.  Ascertain  from  the  contractor  the 
impact  of  these  changes  on  cost  and/or  schedule.  Report  code  growth  and 
impact  to  the  buying  office. 

(b)  Audit  for  compliance  with  DODI  5000.31,  5000. 5X  and  AF  directives  on 
tne  use  of  High  Order  Languages  <HOLs)  and  Instruction  Set  Architectures 
(ISAs). 

(!)  Monitor  software  and  documentation  to  identify  proprietary  products 
used  anywhere  in  the  development  process.  Report  items  to  the  buying  office 
wnicn  are  incorrectly  identified  as  proprietary  in  accordance  with  the 
contract . 

(8)  Monitor  processes  used  to  assure  concurrent  review/approval  of 
interrelated  hardware  and  software  elements  of  a  system. 

(9)  Monitor  tne  contractor’s  use  of  structured  programming  techniques. 

(10)  Assess  tne  contractor's  readiness  for  scheduled  reviews  and  audits 
and  provide  timely  input  to  the  program  office. 
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ill)  Assure  contractor  nas  identified  ail  support  software  tools  and 
verified  that  each  tool's  data  rights  designation  is  in  accordance  with 
contract’s  data  rights  clauses. 
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r  .gure  -Sr,. i-  INWr.ri  UK  Ciir.'CKUUT  -  SOKTWAHF  oUAi.JTY  ASSUBANCf 


i-ei-  T  j c _ MhtiC'i'UriA'i  ci  OF  ENG  INcifc.fi  ING  &  TEST  .JANUARY  1982 


Arh  duo- 1 A 
«  r  o  0  H  /  4  —  1 
Ar'h  7  A  —  1 


MlL-S-t.<?7?9A 
MlL-3"’Li-Ib79 
MiL-STu-AgG 
MIu-STu-Ad j 


■  '■••OtSfdHi  Jffic--  .siii'tw.ir^  Quality  Assurance  Management 

•i.  „s  someone  witniu  tn«  program  office  i  P  ul  i  designated  as  being  tne 
e  r ry  ir..:  l vidua  1  responsible  for  Software  wuality  Assurance  vSQA )? 

o..'-,  responsibilities  included  in  ] oi  description  for  either  CA 
;  >-rs.i;.:.t  1  computer  res  juree  personnel? 

j.m  covered  i:.  -any  of  tne  Pu’s  operating  instructions? 

i:..„ivi  iua.:>  responsible  for  SvA  nave  any  background  or  training 
..  -ftwar-  ::i  gtnerval .  t  30 A  specifically? 


r.  Is  .>»?.  i.uc .  ««:e<5  in  any  ongoing  training  programs  for  appropriate  P0 


.  .  : r -e-contract  software  Quality  Assurance 

i.  Has  ail  program  management  ana  policy  direction  been  reviewed  witn 
res-u'ot  to  ogA  t 

:i  i anv  .jjn  requirements  been  incorporated  into  tne  PMP?  Were  tnev 
i-  .  .  -.oo-  iinati'Jt-  witt.  other  software  related  functions  of  tne  PC? 

.  was  Ml  c-i. -- -'77gfi ,  and/or  tne  quality  assurance  portions  of 
Mu. -or..'- it? a  invoiced  for  tnis  contract?  If  so,  now  were  the  requirements 
tailored? 


d.  is  tnere  a  specific  procedure  to  assure  SwA  requirements  nave  been 

i ricorporated  into  tne  contract  or  HFP  (.specifications,  SOW,  required  reports 
on  Cub..,  etc.,'? 

e.  Are  tnere  >wA  evaluation  criteria  established  for  use  ir.  source 

sg  set  i  on? 


is  anyone  specifically  responsible  for  SC A  considerations  during 
pre c-intraotual  activities  such  as  acquisition  strategy  panels,  source 
•j-ii1- evaluation,  worn  statement  reviews,  pre-award  surveys,  etc? 

.■ .  f  :  rmware  treated  as  software? 

n.  Are  nigh-  •••i«r  language  requirements  of  D0D1  5000. 31  arid  AFP  P00-1A, 
V-i .  !  - .  A  CSC  gup  1,  met.  or  nave  appropriate  waivers  been  obtained? 


76 


HELPS,  Volume  II,  Revision  1 
Contract  Management 


1  December 


1j8; 


3.  Early  Stage-Contract  QA  Planning/Action 

a.  Is  SQA  covered  in  any  Memorandum  of  Agreement  (MuA)  or  Oualitv  Assurance 
Letter  of  Instruction  (QALI)? 

b.  Has  the  ability  of  the  CAO  to  monitor  SQA  activities  of  the  contractor  neeri 
reviewed?  Is  required  training  or  hiring  being  done? 

c.  Have  the  contractors  plans/ procedures  been  reviewed  to  assure 
implementation  of  SQA  requirements? 

d.  Have  SQA  requirements  been  imposed  on  subcontractors? 

e.  Has  the  ability  of  the  contractor  to  monitor  any  necessary  sut -contractor 
SQA  activities  been  verified? 

4.  Contract  Performance  and  Contract  Monitoring 

a.  Have  specific  SQA  considerations  been  included  in  the  topics  to  be  covered 
at  reviews  and  audits  (SDR,  PDR,  CDR,  FCA,  PCA)  or  have  any  SQA  specific  reviews 
been  planned  or  accomplished? 

b.  Do  SQA  personnel  participate  in  formal  reviews  and  audits  (SDR,  PDR,  .'DR, 

ECA,  PCA)? 

c.  Do  tbe  CAO  and  the  PO  monitor  the  status  of  software  problem, 'trouble 
reports  and  software-related  ECP’s? 

d.  Is  there  a  specific  method  for  monitoring  overall  effectiveness  of  trie 
contractor's  SQA  program/efforts? 

e.  Are  tnere  documented  procedures  by  which  PO  SQA  personnel  can  review  and 
discuss  software  deficiencies  with  the  CAO? 

f.  Are  there  documented  procedures  for  requiring  the  contractor  to  correct 

software  def iciencies? 

g.  What  actions  are  taken  by  PO  SQA  personnel  to  assure  the  contractor  is 
pursuing  adequate  corrective  action  with  respect  to  software? 

n.  Are  SQA  personnel  assisting  the  CRWG  and  the  planned  support  organi  imt.  :oti 
(AFLC  or  using  command)  in  developing  SQA  requirements  to  be  met  after  t rogram 
management  responsibility  transfer  (PMRT)? 

i.  Have  appropriate  SQA  requirements  been  included  in  the  CRISP  by  tne  ’RWC? 

j.  Are  SQA  personnel  involved  in  the  QA  assessments  for  major  programs  tr.at 
are  required  by  AFR  74-1  and  AFSCR  74-1? 

k.  Is  someone  in  the  PO  designated  as  being  responsible  for  reviewing 
verification/testing  oross-reference  listings  for  software  to  assure  the  contractor 
will  test  all  software  requirements  appropriately ? 
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1.  are  JwA  personnel  involved  in  reviewing  test  procedures? 

in.  Do  5WA  personnel  participate  in  formal  testing  (qualification  testing, 

DT&el,  iuT&K)? 

n.  are  i>wA  personnel  involved  in  reviewing  software  documentation 
.(specifications,  manuals,  listings,  etc.}  for  compliance  and/or  acceptance  purposes? 

o.  Are  tnere  provisions  for  government  access  to  software-related  data  and 
information  used  jy  the  contractor  during  development? 
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Figure  '  -9-3*  Software  Quality  Assurance  Plan  (SQAP) 

Checklist 

Def i ni tion :  Tne  SQAP  describes  the  Software  Quality  Assurance  (SQA)  Program 
organization  and  procedures  of  the  contractor  to  assure  that  software 
delivered  under  the  contract  complies  with  the  requirements  of  tne  contract. 
The  SQAP  shall  be  directed  toward  the  design  and  production  of  software  that 
is  reliable,  maintainable,  and  functional.  This  SQAP  is  designed  1  Aw 
DI-F-3521.  The  SqAP  can  be  incorporated  in  the  CPDP  and  is  required  as  a 
proposal  document. 

1.  Does  the  SQAP  plan  identify  the  organizational  element's)  responsible 
for  various  software  quality  efforts  to  be  accomplished? 

2.  Do  the  software  quality  personnel  performing  the  necessary  functions 
have  sufficient  authority,  responsibility,  and  freedom  of  action  to  evaluate 
software  design  and  product  activities,  and  t>.  initiate  and/or  recommend 
changes? 

3.  Do  tne  duties  to  be  performed  oy  tne  software  quality  personnel  have 
specific  documented  definitions? 

k.  Does  tne  SQAP  reflect  all  software  development  information  or  is  it 
clearly  cross-referenced  to  other  contractual  documentation?  Is  such 
information  clearly  referenced  by  title,  source,  and  paragraph  number?  Is 
there  a  cross-reference  table  which  snail  clearly  indicate  the  correspondence 
between  this  plan's  paragraph  organization  ano  the  requirements  of  tne  format 
ir,  accordance  with  DI-fi-3521  or  other  approved  t  ID? 

5.  Is  the  scope  of  the  SQA  plan  provided  and  are  all  development 
policies,  procedures,  and  products  subject  to  the  provisions  of  the  plan 
described? 

b.  Are  tne  objectives  of  the  SQA  Fn an  and  the  goals  of  the  SQA  efforts 
stated  using  software  development  milestones  and  visible  indicators? 

7.  is  there  a  list  of  applicable  standards,  and  specifications  to  be 
fol lowed? 


td.  Does  the  contractor  identify  the  tools,  techniques,  methodologies,  and 
records  that  shall  be  employed  in  the  performance  of  work  which  will  support 
SiqA  objectives  and  goals,  and  describe  how  their  use  will  augment  or  satisfy 
the  requirements  of  M1L-S-52779A? 

9.  Does  the  plan  reference  or  document  the  policies,  practices,  and 
procedures  by  which  design  documentation  is  reviewed  to  evaluate  design  logic, 
fulfillment  of  requirements,  completeness,  and  compliance  wiih  specific 
standards? 

10.  Is  it  noted  that  design  shall  be  subjected  to  independent  review  prior 
to  its  release  for  coding? 
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11.  Does  the  plan  reference  or  document  the  contractor’s  policies, 
practices  and  procedures  for  normally  approving  or  certifying  the  description, 
authorization,  and  completion  of  work  performed  under  the  contract? 

12.  Does  the  plan  provide  for  effective  monitoring  to  assure  compliance 
with  these  procedures? 

13*  Does  the  plan  reference  or  document  the  contractor's  policies, 
practices,  and  procedures  for  the  controlling  and  handling  of  source  code  and 
object  code  and  related  data  in  their  various  forms  and  versions,  from  the 
time  of  their  initial  approval  or  acoeptance  until  they  have  been  incorporated 
into  the  final  media?  (The  objective  of  these  controls  is  to  ensure  that 
different  computer  program  versions  are  accurately  identified  and  documented, 
that  no  unauthorized  modifications  are  made,  that  all  approved  modifications 
are  properly  incorporated,  and  that  software  submitted  for  testing  is  the 
correct  version). 

14.  Does  the  plan  reference  the  documentation  standards  and  programming 
conventions  and  practices  to  be  used  for  all  software  referenced  or  documented 
in  tne  plan? 

15-  Does  the  plan  reference  a  library  procedure  or  are  they  incorporated 
within  the  plan? 

16.  Docs  the  plan  reference  or  document  the  contractor's  policies, 
practices,  ar.d  procedures  for  preparation  and  execution  of  formal  and  in-house 
reviews  and  audits,  for  establishing  the  traceability  of  initial  contract 
requirements  through  the  successive  baselines,  and  for  ensuring  that  the 
reviews  and  audits  are  conducted  in .accordance  with  the  prescribed  policies, 
practices,  and  procedures? 

17.  Is  the  schedule  for  reviews  and  audits  referenced,  or  stated  as  in 
attaenment  to  the  plan? 

Id.  Does  tne  plan  specify  the  relationships  between  the  SQA  and  (CM) 

Conf iguration  Management  programs  and  reference  or  document  the  policies, 
practices,  and  procedures  for  assuring  that  the  objectives  of  the  CM  program 
are  being  attained? 

19-  Does  the  plan  reference  or  document  policies,  practices,  and 
procedures  for  aaauring  the  accomplishment  of  the  following: 

a.  Design  of  software  to  allow  the  complete  testing  of  all  programs 
and  subprograms. 

b.  Review  of  test  requirements  and  criteria  for  adequacy, 
feasibility,  and  traceability  and  satisfaction  of  requirements. 

c.  Review  of  test  plans,  policies,  practices,  procedures,  and 
specifications  for  compliance  with  contractor  and  contractual  requirements  and 
to  insure  that  all  authorized  and  only  authorized  changes  are  implemented. 
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cl.  Verification  that  tests  are  conducted  in  accordance  with  approved 
test  plans  and  procedures. 

<•.  Certification  that  test  results  are  the  actual  findings  of  the 

tests. 

Review  and  certification  of  test  reports. 

g.  Ensuring  that  test  related  media  and  documentation  are  maintained 
to  allow  repeatability  of  tests. 

h.  Ensuring  that  support  software  and  computer  hardware  to  be  used  to 
dtvi’Icp  ;  nd  test  software  and  hardware  under  the  contract  are  acceptable  to 

the  Gc  vc:  .vent. 

PD.  Poes  the  plan  reference  or  document  policies,  practices,  and 
procedure s  which  assure  the  prompt  detection,  documentation,  and  correction  of 
software  problems  and  oeficiencies ?  Do  the  procedures  include: 

i  .  Documenting  and  reporting  problems  and  deficiencies  to  appropriate 

manage.:.*!  t  levels. 

t .  Analysis  of  data  and  examination  of  problem  and  deficiency  reports 
to  cietcrrine  their  extent  and  causes. 

c.  Review  of  corrective  measures  to  insure  that  problems  ar.d 
deficiencies  have  been  resolved  anc  correctly  reflected  In  the  appropriate 

doc-i-’-.r  r.ts. 

d.  Analysis  of  trends  in  performance  of  work  to  prevent  the 
development  of  noncompiiant  products. 

e.  Analysis  or  review  as  otherwise  provided  for  in  the  contract. 

21.  Poes  the  plan  reference  or  document  the  policies,  practices,  and 
procedures  to  assure  that  all  software  acquired  from  subcontractors  conform  to 
applicable  requirements  of  the  contract? 
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Figure  II-9-4.  GUIDANCE  FOR  APPLYING  MIL-S-52779A 

Tne  following  pages  provide  guidance  for  applying  and  tailoring  M11-S-52779A, 
Software  Quality  Assurance  Program  Requirements,  to  contracts.  The  first  page 
presents  guidance  for  cnooslng  which  paragraphs  of  the  MIL-S  apply  to 
non-dellverable  software.  The  pages  following  It  give  a  suggested  Statement 
of  Work  (SOW)  paragraph  and  "applicable  documents"  tailoring  guide  for  the  SOW. 

Close  examination  of  the  tailoring  will  also  reveal  that  some  requirements  of 
tne  MIL-S  nave  been  expanded  to  give  a  more  comprenenslve  SQA  program.  An 
example  of  this  Is  the  tailoring  of  paragraph  3.2.4,  Documentation,  to  require 
tne  contractor's  SQA  to  review  Internal  software  development  documentation, 
suon  as  programmer  notebooks,  and  all  deliverable  software-related 
documentation. 

In  using  tnls  guide,  do  not  forget  that  It  Is  only  a  guide  and  must  be  applied 
Judiciously  to  meet  the  requirements  of  a  given  system  acquisition  while 
giving  tne  government  tne  maximum  cost-effective  contractor  SQA  program. 
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GUIDANCE  FOR  APPLYING  MIL-S-52779A  TO  NON-DELIVERABLE  SOFTWARE 


for  the  purposes  of  applying  MIL-S-52779A  to  non-deliverable  software,  it  is 
necessary  to  categorize  this  software  into  two  classifications: 

1.  Test  software.  Automated  Test  System  (ATS)  and  Computer  Aided  Inspection 
used  to  accept  or  test  CPCI/C.I,  total  system,  subsystem,  or  lower,  whether 
used  for  in-process  testing  or  final  acceptance.  Suggested  minimum  applicable 
paragraphs  of  MIL-S-52779A  are:  Sections  1  &  2,  Paragraphs  3.1,  3.2,  3-2.1, 
3.2.2,  3.2.4,  3-2.5,  3.2.7,  3.2.8,  3.2.9,  and  3-3  and  Section  4. 

2.  Other  non-deliverable  software.  This  includes  such  software  as  support 
tools  (e.g.  simulators,  emulators,  test  case  generators,  code  auditors,  etc.), 
computer  aided  design  (CAD)  software  used  to  design  or  support  the  design  of 
software  or  equipment  (e.g.  automated  drawings,  design  analysis  tools, 
modeling  tools,  digitizers,  data  base  design  aids,  etc.),  computer  aided 
manufacturing  (CAM)  software  and  software  to  control  robotics  and  numerical 
control  processes.  Suggested  minimum  applicable  paragraphs  of  MIL-S-542779A 
are:  Sections  I  &  2  and  Paragraphs  3-2,  3.2.1,  3.2.5,  3.2.7,  3-2.9,  and  3-3. 

This  may  be  represented  tabularly  as  follows: 


Other  Non- 


MIL-S-52779A 

Test  Software 

Deliverable  Software 

Sect 

1  A  2 

X 

X 

Para 

3-1 

X 

Pa  ra 

3-2 

X 

X 

Para 

3.2.1 

X 

X 

Para 

3.2.2 

X 

Para 

3.2.3 

Para 

3.2.4 

X 

Para 

3.2.5 

X 

X 

Para 

3.2.b 

Para 

3-2.7 

X 

X 

Para 

3.2.8 

X 

Pa  ra 

3.2.9 

X 

X 

Para 

3. 3 

X 

X 

* 

a 

X 
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•in is  sample  aLds  in  composing  the  Statement  of  Work  and  is  intended  to  3erve 
•as  guidance  in  tailoring  MIL-S-52779A,  Software  Quality  Assurance  Program 
•acquirements.  It  may  be  included  in  either  the  Quality  Assurance  or  the 
•software  Development  task  of  tne  Statement  of  Work,  as  appropriate,  or  it  may 
*oe  a  "stand-alone"  tasK.  It  is  doubtful  that  all  of  the  model  SOW  would  be 
•used  on  any  one  contract.  Care  snould  oe  used  in  choosing  only  those  para- 
•grapns  tnat  the  program  office  feels  are  necessary. 


X.X.X.  Software  Quality  Assurance.  The  contractor  shall  plan,  develop,  and 
implement  a  comprehensive  software  quality  assurance  (SQA)  progam  IAW 
V;!  L-o-bS 779A  and  tne  following  additional  tasks.  This  program  shall  be 
implemented  through  independent  audit,  review,  and  surveillance  activity  and 
formal  acceptance  of  software  developed  under  contract.  An  SQA  plan  shall  be 
written  to  document  all  aspects  of  tne  SQA  program  and  updates  to  tne  SQA  plan 
will  maue  as  required.  Tne  SQA  Plat)  will  be  delivered  IAW  the  CDRL.  All 
Procedures  and  data  relevant  to  the  implementation  of  the  SQA  program  snail  be 
available  for  government  review. 

a  .  1  .  SyA  Policies  and  Procedures.  Tne  SQA  program  shall  include 
implementation  of  written  policies,  practices  and  procedures  to  assure  that 
software  meets  ail  contract  requirements.  Tnese  documents  shall  oe  available 
for  re j Lew  by  government  personnel. 

X.X.X.i.  software  Program  hequi remerit s.  The  SQA  program  snail  address  tne 
fo. lowing  requirements. 

X.X.X.f.l.  Tools,  Techniques,  and  Methodologies.  Tne  contractor  shall  review 
all  tools,  techniques,  and  methodologies  prior  to  use  for  the  design, 
devel  >pment,  testing  or  quality  assurance  of  any  software.  The  review  snail 
ce  for  completeness,  adequacy,  evidence  of  validation,  applicability,  and 
adherence  tr-  good  software  design/development  practices  as  required  by  this 
contract  and  in-house  standards.  Any  deficiencies  snail  be  recorded  and 
brought  to  tne  attention  of  tne  software  development  and  project  managers  and 
promptly  corrected 

X.X.X.. Computer  Program  Design  and  Code.  The  contractor  snail  conduct 
incremental  reviews  of  tne  design  and  design  documentation  starting  at  the 
earliest  point  in  tne  design  process.  Tne  review  process  snail  include  but 
not  i>e  limited  to  tne  following: 

X  .  X .  X 1  Preliminary  Design  Pnase. 

a.  heview  the  overall  software  system  configuration  to  determine  whether 
computer  program  conf iguration  item  (CPCI)  and  computer  program  component 
iCtCJ  oreaKdowr.  is  appropriate  and  logical  considering  tne  major  functions  of 
tne  system. 

o.  heview  tne  Computer  Program  Development  Specification  for  completeness 
of  purpose  and  description.  Verify  that  peformance,  design,  memory,  timing, 
interface,  human  performance,  and  data  base  requirements  have  been 
aoeomp 1 i shed . 
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c.  Review  allocation  of  Computer  Program  Development  Specification 
requirements  to  CPC's  and  verify  traceability  down  to  the  module  level. 

d.  Review  the  Verification  Cross-Reference  Matrix  of  tne  Computer  urogram 
Development  Specification  to  assure  that  all  requirements  are  tested  and  that 
the  verification  method  is  appropriate. 

e.  Review  Preliminary  Design  Review  ( FDR )  data  and  attend  PDR.  Identify 
deficiencies  and  follow-up  on  corrective  action  to  assure  appropriateness  ir.<: 

implementation  of  resolution. 

X.X.X.2.2.2.  Detailed  Design  Phase. 

a.  Accomplish  tne  items  of  paragraph  X.X.X.2.2.1  except  for  subparagraph 
d  and  e.  Substitute  preliminary  "Computer  Program  Product  Specification"  fr 
"Computer  Program  Development  Specification"  and  "Critical  Design  Review" 

(CDR)  for  "PDR. " 

b.  Review  the  test  approach  and  methodology.  Verify  that  test  plans  are 
comprehensive  by  insuring  that  all  specif ication  requirements  will  be 
verified.  Verify  tne  adequacy  of  acceptance  criteria.  Review  tne  Test 
Plan/Procedure  Cross  Reference  Index  of  the  Computer  Program  Product 

Specif ication  for  completeness  and  adequacy. 

c.  Tne  contractor  snail  review  subsequent  releases  of  design 
specifications  for  completeness  and  traceability  (as  described  above  in 
X.X.X.2.2.1  b  and  c  for  computer  program  specifications). 

X.X.X.2.2.3*  Coding.  Programming  standards  and  conventions  shall  tie 
documented  and  available  for  review  by  the  government  personnel.  Contractor 
shall  systematically  sample  and  examine  coding  to  ensure  compliance  with 
requirements,  design  and  programming  standards  and  conventions.  Deficiencies 
shall  be  brought  to  the  attention  of  the  appropriate  level  of  management, 
corrected,  and  tracked  IAW  MIL-S-52779A,  paragraph  j>.2. 9. 

X.X.X.2.3.  Work  Certification.  Contractor  snail  monitor  the  procedures  for 
formally  approving  or  certifying  the  description,  authorization,  arid 
completion  of  work  performed  under  this  contract  ard  assure  compliance  with 

these  procedures. 

X.X.X.2.R.  Documentation. 

Contractor  snail  assure  review  of  all  software-related  documentation  to  assure 
compliance  with  all  requirements  of  the  contract  ami  applicable  in-house 
standards. 

X.X.X.2.4.1  Software  Development  Documentation.  Contractor  bwA  shall 
per- lodica  1  ly  audit  3oftwar-e  development  documentation  to  assure  compliant- 
with  applicable  m-nouse  standards  and  procedures.  Ibis  imiit  sn all  u-ie 

as  a  minimum:  assurance  of  proper  design  and  test  ln'a,  :iec,-.;,Ra.  v  ..  , 

maintenance  of  status  logs  (for  history,  changes,  and  ornn ems  i  , 
pronlero  reports,  and  programmer  .notebooKS,  unit,  development  let  :  *  •• 

bn  umetita:  1  ■ » r i  :.m'  ir-'-  :  n  use. 
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X . .  ‘i . ,  U;l ;  v-t-ai,- le  documentation.  in  addition  to  the  specification 
rev  lew  requirement:-  of  paragraph  X.X.X.g.,c,  the  contractor  snail  assure  review 
of  deliverable  software  related  documents  for  compliance  with  contractual 
requirements  and  applicable  in-house  standards  and  procedures.  Examples  of 
sol twure-re 1 ated  documents  are  specifications,  data-base  design  documents, 
interface  specifications,  positional  handbooks,  users  manuals,  operator 
manuals,  ana  maintenance  manuals. 

X.X.X.^.n.  Computer  Program  i.i  nrary  Controls.  The  Contractor  shall  audit  for 
comp li m De  with  tne  procedures  and  controls  for  the  Computer  Program  Library. 

a.X.a.a'.c  reviews  ana  Audits.  The  contractor  snail  address  both 
internal /’in forma  I  and  formal  reviews  and  audits  (to  include  internal  design 
documentation  reviews  and  design  and  code  walkthroughs )  and  prepare  a  schedule 
P  irticipation  1AW  to  paragraph  i.2.b  of  MIL-S-52779 A. 

X. X. a.  .  j .  Configuration  Management  (CM).  Contractor  snail  ensure  that 
..iie)u-.te  oonflgurati  .it:  management  techniques  and  proceaures  exist  to  control 
software  configurations.  Tnese  snail  address  both  tne  contractor's  internal 
i-velopment  environment  and  formal  baselines  reportable  to  tne  Government. 

o  .u, tractor  anal  1  define  tne  responsibilities  for  controlling  various 
1  ••■."■■fries  and  media  te.g.  working  library  and  master  media)  and  other 

igurati-Jh  management  procedures  as  applicable  to  SUA,  software  development 
and  configuration  management  personnel.  Contractor  shall  audit  for  compliance 
with  tnese  techniques  ind  procedures. 

X.X.x.r..d.  jesting.  Tne  contractor  shall  ensure  that  support  software  and 
hardware  used  to  develop  and  test  software  and  hardware  meets  the  requirements 
of  trie  contract  and  all  in-house  standards.  Tne  support  software  and  hardware 
shall  also  be  reviewed  prior  to  use  of  evidence  for  validation  testing  under 
this  contract  or  from  previous  applications. 

X.X.X.2.9.  Corrective  Action.  The  Contractor  shall  assure  that  documented 
prone  lures  exist  and  are  properly  complied  with  which  assure  prompt  detection, 
documentation  and  correction/disposition  of  software  problems  and  deficiencies. 

X . X . X . a  Applicable  uocuments 


Number  and  rate  i'itle 


Application 


M  i  L-b  -‘.9V7',‘ A 
1  Aug  79 


Software  Quality  Assurance  All,  para  1.1. 

Program  Hequ irements  tailored  is  by  the 

following 


Paragraph  1.1.  Add  the  following:  All  software  tnat  is  modified  under  this 
contract  will  be  treated  as  software  in  respect  to  the  SQn  program. 
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lOlMTIFtCATlON  NOIJl 


Software  Quality  Assurance  Plan 


USAF  DI-R-3521 


A  DUCDWTIWnjWOU 


3.1  This  Data  Item  Description  (DID)  defines  the 
requirement  for  a  Software  Quality  Assurance  (SQA)  plan. 


4.  APPROVAC  OATS 

1982  September  21 


S.  OPPICf  ON  ANiUAAT 
RUPONRNUTf 

HQ  MAC/ AD 


«.  OOC  PCOUIRCO 


ft.  WM OVAC  LIWtATlOH 


•mm MCftriOM/fM 


7.1  This  DID  may  be  applied  to  the  acquisition  of 
deliverable  software  and  firmware,  either  alone  or  as  a 
portion  o-f  a  svstem  or  subsystem,  and  to  non-deliverable 
design,  test,  support,  and  operational  software  developed 
under  contract. 

7.2  This  DID  satisfies  the  requirements  of 
Paragraphs  3.2  and  3.3  of  MIL-S-52779A,  Software  Quality 
Assurance  Program  Requirements 


Meat.  MUMftftma 

OMB  Exempt 
*AMSC  No.  A3081 


ftKftTMM  INiriUC  TIOMI 


10.1  The  SOA  Plan  shal L  include  sections  that  specifically  address  each  ol  the 
following  paragraphs  of  MIL-S-527 79A: 


a.  3. 

b.  3. 

c.  3. 

d.  3. 

e.  3. 

f.  3. 


i.  3.: 

j.  3.: 

k.  3.: 


Software  QA  Program  Requirements 
Tools,  Techniques,  and  Methodologies 
Computer  Program  Design 
Work  Certification 
Document at  ion 

Computer  Program  Library  Controls  (CM) 
Reviews  and  Audits 
Configuration  Management 
Testing 

Corrective  Action 
Subcontractor  Control 


10.2  The  plan  may  be  prepared  in  the  vendor's  format,  subject  to  the  following 

constraints : 


10.2.1  It  shall  be  typewritten  or  clearly  lettered  and  copies  shall  be  reproduced 
with  non-fading  ink  on  white  8%”  x  11”  paper. 

10.2.2  The  tl-le,  date,  and  contract  number  shall  be  on  a  title  page.  Other  relevant 
Information  may  be  included. 

j.G.2.3  Att ichments  shall  be  prepared  or.  standard  letter  size  paper  or  standard  size 
engineering  drawing  paper.  Each  attachment  shall  be  fullv  identified  and  snail  he 
referenced  in  the  text  of  the  plan. 
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Section  10.  Computer  Program  Development  Capability/Capacity  Review 

(Pre-Award  Survey) 

10-1.  Review  the  AFSC  Sup  to  AFR  800-14,  Vol  I. 

10-2.  Review  AFSCR  70-7. 

10-3.  Review  any  corporate  software  documents  available,  i.e.,  Software 
Quality  Plans,  Computer  Development  Plans/Procedures. 

10-4.  Review  Figure  II-10-1,  Computer  Program  Development  Capability/Capacity 
Review  Questions. 

NOTE :  The  Manufacturing  Directorate  within  ESD/AL  has  primary  responsibility 
for  overseeing  pre-award  surveys  and  manufacturing  management  production 
capability  reviews.  Both  areas  revolve  around  evaluation  of  offeror’s  in  the 
source  selection  time  frame.  Any  pre-award  survey  effort  should  be 
coordinated  with  this  organization. 
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f.  Interface 

1.  How  does  software  engineering  interface  with  systems  engineering? 

2.  How  does  functional  and  subsystem  engineering 
interface/participate  in  software  engineering? 

3.  How  does  software  development  interface  with  configuration 
management,  data  management,  test  and  quality  assurance? 

g.  Reporting 

1.  How  is  status  reporting  correlated  among  cost  performance 
reporting,  technical  documentation  and  program  management  status  reporting? 

2.  Who  is  responsible  to  make  sure  all  software  reporting  is 
consistent  across  disciplines  and  subsystems? 

3.  Who  is  responsible  to  determine  and  report  software  status? 

h.  Staffing 

1.  How  does  the  lead  software  manager  determine  and  acquire  the 
personnel  resources  needed? 

2.  How  does  the  lead  software  manager  acquire  additional  resources? 

3.  How  is  the  software  manager  assured  of  retaining  the  personnel 
resources  and  expertise  for  the  duration  of  the  project? 

4.  How  long  are  the  key  software  developers  committed  to  the  program 
team?  How  often  is  this  reviewed? 

5.  Are  there  any  significant  changes  in  your  software  development 
staffing  for  PDR ,  CDR  or  during  the  test  phases? 

6.  What  is  your  criteria  for  bringing  on  job  shopper  software 
specialists? 

3.  AVAILABILITY  OF  SOFTWARE  PERSONNEL 


a.  Requirements 

1.  How  do  you  determine  the  number  of  software  personnel  required  to 
initiate  a  program? 

2.  What  is  the  basis  for  the  sizing  of  the  manpower  requirement? 

b.  Availability  Criteria 

1.  What  is  your  criteria  for  determining  availability,  i.e.,  bidding 
the  key  software  personnel? 


2.  What  leverage  do  you  have  to  pull  key  people  off  other  programs? 
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c.  Manning  Control  and  Decisions 

1.  Who  participates  in  the  personnel  availability  deliberation  for 
the  program,  and  who  ultimately  decides? 

2.  Are  personnel  resources  controlled  out  of  a  "home  office",  i.e., 
in  a  matrixed  organization?  If  so,  how  will  that  affect  the  assignment  of 
personnel  to  this  project,  and  the  control  of  personnel  assigned? 

d.  Incentives 

1.  What  incentives  are  offered  to  key  software  developers  to  motivate 
performance? 

e.  Location 

1.  Do  you  plan  to  develop  all  of  the  subject  program  software  at  one 
geographical  location?  If  not,  describe  the  development  structure  you  intend 
to  use.  Where  does  the  overall  software  development  responsibility  reside? 

1*.  COMPANY  WORKLOAD  PROFILE 


a.  On-going  and  Planned  Contracts 

1.  Identify  all  on-going  and  planned  contracts  which  include  software 
development. 

2.  Identify  the  magnitude  of  the  composite  company  software  efforts 
under  development,  and  planned  efforts. 

3.  Identify  schedules  and  status  of  the  software  development  on  these 
programs. 

b.  Software  Personnel  Profile 

1.  Identify  a  composite  company  profile  of  software  personnel  working 
on  all  on-going  and  planned  contracts. 

2.  Categorize  these  personnel  by  skills  and  experience  and  years  of 
experience,  including  years  with  the  company. 

3.  How  do  you  track  and  manage  personnel  turn-over? 

4.  How  important  is  software  personnel  stability  to  your  program 
stability  and  success?  What  is  the  basis  for  your  assessment  of  stability 
needs? 


5.  Is  software  personnel  stability  dependent  on  the  application 
subsystem  involved? 

c.  Subcontractor/ Job  Shoppers 

1.  How  many  subcontractor  software  personnel  are  employed  on  your 
company  contracts  (workload  profile)? 
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Figure  II-10-1.  Computer  Program  Development  Capability  Capacity 

Review  List  of  Factors 

NOTE:  Tnese  factors  were  developed  by  ASD/EN  under  the  direction  of  Mr.  Phil 
Babel.  Various  draft  iterations  were  commented  on  by  many  organizations, 
including  the  ESD  technical  staff. 

1.  Management  Organization 

2.  Personnel  Qualifications 

3.  Availability  of  Software  Personnel 

4.  Company  Workload  Profile 

5.  Software  Subcontracting 

6.  "Make  or  Buy"  Criteria 

7.  Computer  Program  System  Organization  and  Structure 

8.  Support  Software  Development  Facilities 

9.  Software  Development  Tools 

10.  Software  Management  System 

11.  Software  Cost  Reporting 

12.  Te3t  and  Verification 

13.  Software  Configuration  Management 

14.  Internal  Development  Standards 

15.  Software  Contract  Work  Breakdown  Structure  (CWBS) 

16.  Software  Work  Definition 

17.  Contract  Control  Methods 

18.  Software  Documentation  Approach 

19.  Software  Product  and  Quality  Assurance 

20.  Software  Estimating,  Size,  Schedule  Cost 

21.  IV&V  Interfaces 
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1.  MANAGEMENT  ORGANISATION 

a.  Structure 

1.  How  is  tne  software  development  activity  structured  within  the 
program  <!*•  --lopment  organization? 

>  .  How  is  this  structure  maintained  tnroughout  the  program? 

3.  Where  does  the  overall  software  responsibility  reside?  Is  there  a 
single  lead  softuare  manager  or  are  there  several  lead  software  managers  for 
tne  program? 

4.  Wnat  are  the  minimum  qualifications  for  the  lead  software 
engineering  manager? 

b.  How  are  the  lead  software  manager  and  the  lead  software 
engineering  roles  and  responsibilities  managed? 

b.  How  is  the  total  software  development  team  organized,  top  to 
bottom,  including  intermediate  organizational  and  supervisory  levels? 

b.  Level 

1.  At  what  level  does  software  development  interface  with  program 
management? 

2.  What  is  tne  organizational  level  of  the  lead  person  responsible 
for  software  development? 

3.  At  what  levels  are  the  software  functions  of  program  management 
assigned,  e.g.,  schedule  compliance  and  status  assessment  tracking? 

c.  Control 

1.  Wnat  personnel  resources  are  controlled  by  the  lead  software 

manager? 

2.  Wno  writes  and  controls  tne  lead  software  manager's  performance 

rating ? 

i.  Who  must  agree  and  coordinate  on  software  personnel  requests? 

d.  Completeness 

1.  Wnat  functions  are  performed  by  the  software  development  team, 

e.g.,  analysis  tnrough  test? 

2.  Which  software,  by  oatagory,  is  developed  by  the  software 
development  team? 

1.  Where  is  support  software  developed; 

'■i .  Wnat  percentage  of  the  software  development  team  Ls  ieaieated 

time'; 
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e.  Interface 

1.  How  does  software  engineering  interface  with  systems  engineering? 

2.  How  does  functional  and  subsystem  engineering 
interface/participate  in  software  engineering? 

3.  How  does  software  development  interface  with  configuration 
management,  data  management,  test  and  quality  assurance? 

f.  Reporting 

1.  How  is  status  reporting  correlated  among  cost  performance 
reporting,  technical  documentation  and  program  management  status  reporting? 

2.  Who  is  responsible  to  make  sure  all  software  reporting  is 
consistent  across  dicipline^'  and  subsystems? 

3*  Who  is  responsible  to  determine  and  report  software  status? 

g.  Staffing 

1.  Are  there  software  personnel  matrixes? 

2.  How  does  the  lead  software  manager  determine  and  acquire  the 
personnel  resources  needed? 

3.  How  does  the  lead  software  manager  acquire  additional  resources? 

4.  How  is  the  software  manager  assured  of  retaining  the  personnel 
resources  and  expertise  for  the  duration  of  the  project? 

5.  How  long  are  the  key  software  developers  committed  to  the  program 
team?  How  often  is  this  reviewed? 

6.  Are  there  any  significant  changes  in  your  software  development 
staffing  for  PDR,  CDR  or  during  the  test  phases? 

7.  What  is  your  criteria  for  bringing  on  job  shopper  software 
specialists? 

2.  PERSONNEL  QUALIFICATIONS 
a.  Experience 

1.  What  is  your  corporate  software  development  capability  per  your 
Key  personnel  experience  and  qualifications? 

2.  Identify  the  average  years  of  software  development  experience 
among  your  staff. 

j.  What  is  the  percentage  of  this  experience  acquired  while  with  your 


company? 
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4.  Wnat  personnel  turn-over  rate  are  you  experiencing  in  software? 

b.  Academic  Qualifications 

1.  How  many  of  your  software  development  personnel  have  scientific 
engineering,  math,  or  Computer  Science  degrees?  How  many  of  these  personnel 
will  be  assigned  to  the  subject  program? 

2.  Of  the  proposed  software  staff,  how  many  have  an  advanced  computer 
science  degree? 

3.  How  do  you  determine  the  level  of  academic  background  required  to 
assure  sufficient  technical  capability? 

4.  What  are  your  academic  requirements/standards  for  software 
developers? 

c.  Related  Program  Experience 

1.  What  is  the  relevance  of  the  program  experience  of  your  staff  to 
the  subject  program? 

2.  Does  your  experience  include  real-time  weapon  system  software? 

How  does  that  experience  relate  to  the  needs  of  this  program? 

3.  Unis  factor  should  be  pursued  as  a  function  of  the  subject 
program,  e.g.,  in  electronic  warfare,  experience  in  designing  for 
reprogrammability . } 

d.  Classification  by  Skills 

1.  How  do  you  classify  and  distinguish  software  technical  skills, 

e.g.,  analyst,  engineers,  programmers? 

2.  How  do  you  classify  and  identify  software  development  management 

skills? 


e.  Training 

1.  What  type  of  training  program  have  you  established  for  technical 
and  management  skills  of  software  development? 

2.  Do  you  have  required  in-house  training  programs  for  software 
practitioners? 

3.  How  is  tnis  training  administered,  e.g.,  is  there  a  required 
curriculum  or  set  of  courses? 

4.  Wnat  incentives  exist  for  your  people  to  pursue  the  software 
training? 
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f.  Interface 

1.  How  does  software  engineering  interface  with  systems  engineering? 

2.  How  does  functional  and  subsystem  engineering 
interface/participate  in  software  engineering? 

3.  How  does  software  development  interface  with  conf iguration 
management,  data  management,  test  and  quality  assurance? 

g.  Reporting 

1.  How  is  status  reporting  correlated  among  cost  performance 
reporting,  technical  documentation  and  program  management  status  reporting? 

2.  Who  is  responsible  to  make  sure  all  software  reporting  is 
consistent  across  disciplines  and  subsystems? 

3.  Who  is  responsible  to  determine  and  report  software  status? 

h.  Staffing 

1.  How  does  the  lead  software  manager  determine  and  acquire  the 
personnel  resources  needed? 

2.  How  does  the  lead  software  manager  acquire  additional  resources? 

3.  How  is  the  software  manager  assured  of  retaining  the  personnel 
resources  and  expertise  for  the  duration  of  the  project? 

4.  How  long  are  the  key  software  developers  committed  to  the  program 
team?  How  often  is  this  reviewed? 

5.  Are  there  any  significant  changes  in  your  software  development 
staffing  for  PDR,  CDR  or  during  the  test  phases? 

6.  What  is  your  criteria  for  bringing  on  job  shopper  software 
specialists? 

3.  AVAILABILITY  OF  SOFTWARE  PERSONNEL 


a.  Requirements 

1.  How  do  you  determine  the  number  of  software  personnel  required  to 
initiate  a  program? 

2.  What  is  the  basis  for  the  sizing  of  the  manpower  requirement? 

b.  Availability  Criteria 

1.  What  is  your  criteria  for  determining  availability,  i.e.,  bidding 
the  key  software  personnel? 

2.  What  leverage  do  you  have  to  pull  key  people  off  other  programs? 
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c.  Manning  Control  and  Decisions 

!•  Who  participates  in  the  personnel  availability  deliberation  for 
tne  program,  and  who  ultimately  decides"? 

2.  Are  personnel  resources  controlled  out  of  a  "nome  office",  i,e., 
m  a  matrixes  organization?  If  so,  how  will  tnat  affect  the  assignment  of 
personnel  to  this  project,  and  tne  control  of  personnel  assigned? 

d.  Incentives 

1.  Wnat  incentives  are  offered  to  key  software  developers  to  motivate 
performance? 

e.  i-ocation 

1.  Do  you  plan  to  develop  all  of  the  subject  program  software  at  one 
geographical  location?  If  not,  describe  the  development  structure  you  intend 
to  use.  Where  does  tne  overall  software  development  responsibility  reside? 

4,  CUMPANY  WORKLOAD  PROFILE 


a.  Un-going  and  Planned  Contracts 

1.  loentify  all  on-going  and  planned  contracts  which  include  software 
development. 

2.  Identify  the  magnitude  of  the  composite  company  software  efforts 
under  development,  and  planned  efforts. 

3.  Identify  schedules  and  status  of  the  software  development  on  these 
programs. 

b.  Software  Personnel  Profile 

1.  Identify  a  composite  company  profile  of  software  personnel  .-rrking 
on  all  on-going  and  planned  contracts. 

2.  Categorize  these,  personnel  by  skills  and  experience  and  years  of 
experience,  including  years  with  the  company. 

3.  How  do  you  track  and  manage  personnel  turn-over? 

4.  How  important  is  software  personnel  stability  to  your  program 
stability  and  success?  Wnat  is  the  basis  for  your  assessment  of  stability 
needs? 


b.  is  software  personnel  stability  dependent  on  tne  application 
subsystem  involved? 

c.  Subcontractor/ Job  Shoppers 

1.  How  many  subcontractor  software  personnel  are  employed  on  your 
company  contracts  (workload  profile)? 
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2.  How  many  job  shoppers  are  employed  on  your  programs? 

d.  Financial  Profile 

1.  Wnat  is  your  balance  sneet  status  -  assets/liabilities/net  worth? 

2.  Do  you  have  a  copy  of  your  last  external  audit  report? 

3.  Wnat  is  the  cost  structure  of  your  software  business  component  by: 

direct  salaries 
fringe  benefits 
pensions 
other  overhead 
overhead  rate 

*4.  What  have  been  your  software  contract  profiles  (sales)  over  the 
last  five  years? 

5.  Identify  the  contract  dollars  by  type  of  software  development, 
i.e.,  business,  command  and  control,  real-time,  etc. 

5.  SOFTWARE  SUBCONTRACTING 

a.  Management 

1.  How  do  you  set  up  to  manage  subcontractors  developing  software? 

2.  Who  in  your  company  is  responsible  for  their  contractual  software 
performance? 

3-  How  does  the  program  director  manage/control  this  software? 

4.  Do  you  apply  you r  software  development  standards  and  tools  to  your 
subcontractors?  If  not,  what  are  your  requirements  to  assure  the 
subcontractor  adheres  to  standards? 

5.  How  do  you  assess  (survey)  a  subcontractor ' s  software  development 
capabilities  prior  to  selecting  a  specific  subcontractor? 

b.  Reporting 

1.  Wnat  reporting  do  you  require  of  your  subcontractor  relative  to 
software?  How  is  this  reporting  tied  to  the  subcontractor  WBS?  To  what  leve, 
do  you  require  the  subcontractor's  software  to  be  identified  and  reported  in 
his  WBS? 

2.  Provide  samples  of  specific  reports  from  on-going  subcontracts. 

3.  Specifically  how  do  you  assess  a  subcontractors  software  (SW) 
status?  What  do  you  base  your  assessment  on? 


4.  Do  you  require  a  detailed  report? 
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c.  Interfaces 

1.  How  is  tne  subcontractor's  software  interfaced  with  your  internal 
software  ana  that  of  other  subcontractors? 

2.  How  ao  you  interface  the  various  subcontractors  software  technical 
requirements? 

3-  Wnat  approach  do  you  follow  to  allocate  computer  program 
development  requirements  to  subcontractors? 

d.  Contractual 

1.  Ho  do  you  establish  your  contract  terms,  internal  standards? 

2.  What  are  your  rights  to  software  clauses  and  criteria?  Is  it 
possible  a  subcontractor  will  deliver  software  bearing  a  proprietary  data 
legend?  If  so,  how  do  you  propose  to  satisfy  the  Government's  rights  to  data 
requirement? 


3.  Have  you  established  contractual  milestones? 

4.  Do  you  provide  software  development  award  fees  or  contractual 
incentives? 


5.  Do  you  set  up  contractual  terms  to  include  progress  payments  hased 
on  software  as  a  line  item  milestone  deliverable? 

e.  Documentation 

1.  Identify  the  set  of  software  documentation  you  require  of 
subcontractors. 

2.  How  do  you  review  and  approve  this  documentation? 

3.  What  is  your  approach  to  documentation  as  criteria  for  milestones? 


f.  Test 


1.  What  are  your  test  criteria  and  procedures  for  accepting 
subcontracted  software? 

b.  "MAKE  OR  BUY"  CRITERIA 


a.  Criteria/Determination  Approach 

1.  What  factors  are  considered  in  determining  whether  or  not  to 
subcontract  for  software  or  subsystem  development? 

b.  Current  and  Planned  Projections 

1.  Do  you  plan  to  use  software  subcontractors  on  the  subject 
contract?  If  so,  identify  the  subsystem/subsystems  functions  to  be 
subcontracted. 
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2.  When  do  you  plan  to  bring  on  the  software  subcontractors? 

c.  Policies 

1.  What  are  your  make  or  buy  policies  for  software  development? 

2.  What  are  your  policies  for  hiring  job  shopper  specialists? 

7.  COMPUTER  PROGRAM  SYSTEM  ORGANIZATION  AND  STRUCTURE 


a.  Organization  (CPCI-Component) 

1.  How  is  your  software  organized,  i.e,  from  top  of  the  computer 
program  system  (CPS)  to  bottom,  i.e.,  individual  components  or  modules? 

2.  Do  you  use  a  tree  structure  to  identify  and  define  this  CPS 
organization?  If  not,  describe  your  approach  to  identifying  (documenting)  the 
top-down  CPS  organizational  structure. 

3-  Is  this  CPS  organization  implemented  via  computer  program  to 
facilitate  update  and  configuration  control?  Describe  your  method  to 
configuration  manage  the  CPS  structure. 

b.  Identification 

1.  How  do  you  assure  that  each  element  of  tne  CPS  is  uniquely 

identified  by  name  and  number? 

2.  How  do  you  assure  that  each  element  of  tne  CPS  is  identified 
consistently  across  all  disciplines,  i.e.,  technical,  management,  contracts, 
cost,  etc.? 

c.  Completeness 

1.  How  do  you  assure  that  all  operational  and  support  functions  are 
allocated  and  accounted  for  in  the  CPS  organization? 

2.  How  do  you  manage  and  integrate  existing  software  with  the  total 
CPS  to  insure  a  complete  system  is  developed? 

d.  Responsibilities 

1.  Is  one  individual  designated  as  responsible  for  all  software 
including  support,  simulation,  and  test  software?  If  not,  how  is  the 
responsibility  delegated? 

2.  If  outside  organizations  are  developing  support  or  other  ancillary 
software,  who  are  they  responsible  to  on  the  program?  How  are  these  efforts 

managed? 


e.  Interfaces 

1.  How  is  the  software  system  interfaced  with  the  firmware  or 
hardware  elements  of  the  subsystems  and  system? 
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2.  How  is  tne  CPS  organized  and  structured  to  include  major 
subcontractor  developed  software? 

6.  SUPPOhT  SOFTWARE  DEVELOPMENT  FACILITIES 

a.  Computer  Systems 

1.  Wnat  is  your  computer  system  nardware  approach  to  software 
development,  i.e.,  identify  your  host  computer  system? 

2.  Identify  tne  computer  nardware  required  to  support  software 
development  on  tne  subject  program. 

3.  Do  you  have  the  host  and  target  computers  available  to  support  tne 
subject  program? 

4.  What  is  your  plan  to  acquire  these  computers? 

5.  Are  you  planning  to  use  any  Government  Furnished  Equipment  (GFE) 
computer  equipment  on  tne  subject  program?  If  so,  identify  these  requirements. 

o.  How  do  you  determine  the  number  of  input  terminals  required 
relative  to  your  peak  programming  need?  Wnat  is  the  expected  turn-around? 

Wnat  is  your  approach  to  acquiring  additional  equipment  if  necessary? 

b.  Support  Software 

1.  Identify  tne  support  software  tools  you  have  which  you  plan  to  use 
on  the  subject  program. 

2.  Identify  additional  tools  required. 

3.  Identify  sources  for  these  tools. 

A.  Are  you  planning  to  use  any  Government  Furnished  Property 
(GFP;/software  tools?  If  so,  identify  tnese  tools. 

c.  Simulation  Facilities 

1.  Do  you  intend  to  use  any  simulation  facilities  in  your  software 
development  process? 

2.  Do  tnese  facilities  exist  or  do  you  plan  to  develop  them? 

3.  Identify  your  approach  to  developing  or  acquiring  these  facilities 
including  software. 

d.  Test  and  Integration  Facilities 

1.  Identify  facilities  you  require  to  perform  the  software 
integration  and  other  systems  integration. 


2.  Do  these  facilities  exist,  if  so,  identify  and  describe  the 
facilities.  Will  these  facilities  be  dedicated  to  the  subject  program? 
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3-  If  the  facilities  do  no  exist,  how  do  you  intend  to  acquire  t be 
required  facilities? 

4.  Do  you  intend  to  use  any  remote  computer  resources  and 
facilities?  How  do  you  plan  to  assure  sufficient  availability  of  these 
resources? 

e.  Deliverable  Support  Systems 

1.  Are  tne  support  software  and  computer  systems  available  for 
delivery  to  the  Air  Force  for  follow-on  software  support? 

2.  Are  any  of  these  resources  proprietary? 

3*  Are  your  subcontractors  support  software  development  resources 
available  for  delivery  to  the  Air  Force? 

4.  How  do  you  nandle  licensing  arrangements? 

9.  SOFTWARE  DEVELOPMENT  TOOLS 

a.  Compiler/ Assemblers/ Linkers /Editors 

1.  Identify  all  software  development  tools  you  intend  to  use  on  the 
subject  program. 

2.  Do  all  these  tools  exist?  Have  these  tools  been  validated?  Do 
you  have  them  Integrated  into  your  software  development  environment? 

3.  If  the  tools  do  not  exist,  what  tools  are  needed  ana  wnat  is  your 
plan  to  acquire  or  develop  the  tools?  Do  you  have  a  back-up  contingency  plan 

b.  Simulations 

1.  Identify  any  simulation  tools  you  intend  to  use  as  part  of  your 
software  development  process. 

2.  Do  you  intend  to  write  any  simulation  software? 

3.  If  you  do,  identify  the  functions  and  use  of  this  software. 

c.  Analysis 

1.  What  is  your  approach  to  system  and  software  requirements  analysi 

2.  Do  you  intend  to  use  any  analysis  tools? 

3.  If  you  do,  identify  these  tools. 

d.  Configuration  Management 

1.  What  is  your  approach  to  software  configuration  management  (CM)? 

2.  Is  any  of  the  configuration  management  process  automated? 
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3.  Do  you  nave  configuration  management  (CM)  software  tools,  e.g., 
CPS  organization  tree? 

4.  Identify  tne  functions  of  your  CM  software  tools  and  provide 
examples  of  tne  products. 

e.  Otner  Tools 

1.  Do  you  nave  software  (SW)  tools  to  support  SW  cost  performance 
reporting,  status  reporting,  Sw  Contract  Work  Breakdown  Structure  (CWBS),  SW 
worK  packages/work  definition,  SW  schedules,  or  SW  documentation? 

2.  Identify  these  tools  and  provide  examples  of  the  products  they 
generate. 


j.  Identify  any  otner  software  development  tools  you  intend  to  use. 

f.  Standards 

1.  Identify  company  standards  which  define  software  development 
tools,  and  use  of  these  tools. 

g.  Computer/Compiler  Standards 

1.  Bo  you  intend  to  develop  the  subject  program  software  using 
acquisition  agency  designated  standards,  e.g.,  JOVIAL  J 7 3 3  and  MIL-STD-1750A? 

2.  Do  you  nave  a  complete  set  of  tools  and  expertise/capability  to 
use  tne  tools? 

io.  software;  management  system 


a.  Computer  Program  Development  Plan  (CPDP) 

1.  Describe  your  use  of  the  CPDP. 

2.  Within  your  corporate  and  program  structure,  which  organization 
writes  tne  CPDP? 

3.  Wnat  information  is  contained  in  the  CPDP? 

4.  Provide  examples  of  your  CPDP. 

5.  Wno  must  coordinate  on  the  CPDP? 
b.  How  often  is  the  CPDP  updated? 

7.  How  are  SW,  CM,  test,  and  QA  planning  handled  in  the  CPDP? 

B.  How  does  tne  CPDP  interface  with  and  integrate: 

(a)  Milestones 

(b)  Detail  Schedules 
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(c)  Work  Definition  Packages 

(d)  SW  Cost  Performance  Reporting 

(e)  CPS  Tree  (Organization) 

b.  Control  Structure  and  Report 

1.  Describe  your  approach  to  software  management. 

2.  How  is  software  controlled?  Do  you  nave  control  tools? 

3.  What  is  your  methodology  (procedural  approach)  to  track  software 

status? 

4.  Do  you  use  a  control  room  approach? 

c.  Internal  Description  -  Standard 

1,  Is  your  software  management  system  described  in  an  internal 
standard  or  document?  Provide  a  copy  of  this  document. 

2.  What  software  management  system  was  used  on  your  three  most  recen 
software  development  programs?  Provide  examples  of  the  management  tools 
applied,  and  products  developed  on  these  programs. 

d.  Schedule  -  Tiers 

1.  Describe  your  approach  to  organizing  and  documenting  (laying  out) 
software  development  schedules  top  to  bottom. 

2.  How  many  tiers  or  levels  of  schedule  are  employed? 

3.  Which  level  is  used  to  status  (track)  the  software  progress  and 
report  status  to  the  acquisition  agency? 

4.  How  do  you  measure  software  status  relative  to  schedule  tracking? 

e.  Internal  Reviews 

1.  Define  your  approach  to  internal  software  reviews. 

2.  Do  you  use  a  milestone  approach? 

3.  What  is  the  basis  for  the  review? 

4.  What  follow-on  actions  are  required  to  correct  deficiencies 
discovered  during  the  review? 

5.  Do  you  use  walk-throughs? 
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o.  wno  are  required  to  participate  in  the  reviews? 

7.  Wnat  specific  criteria  are  used  to  determine  that  the  reviews  are 
successfully  completed? 

f.  Interface 

1.  How  does  software  management  interface  and  integrate  other 
internal  disciplines  such  as  engineering,  configuration  and  data  management , 
cost  performance  reporting,  test,  and  QA? 

2.  How  does  software  management  interface  with  subcontractors 
developing  software? 

3.  How  is  software  management  integrated  with  systems  management, 
nardware  management,  and  firmware  management? 

g.  Development  Tasks 

1.  Define  tne  software  development  tasks  from  start  to  finish,  as 
identified  and  actually  used  within  your  company. 

2.  Are  these  defined  in  a  company  standard? 

3.  How  are  test,  QA,  documentation,  and  CM  integrated  into  these 

tasks? 

n.  System  -  Subsystem  -  Component  Approach 

1.  Do  you  manage/develop  software  within  subsystems  or  across 
subsystems? 

2.  How  are  your  CWBS  and  Work  Definition  Packages  structured? 
i.  Trouble  Reporting 

1.  Describe  your  approach  to  software  trouble/problems  reporting. 

2.  Provide  your  standard  reporting  forms. 

3.  How  are  these  reports  tracked  to  assure;  corrections  of 
deficiencies,  solutions  to  problems,  and  completion  of  tne  effort? 

y.  What  level  within  your  management  organization  reviews  these 

reports? 


5.  Wnat  methods  do  you  use  for  trend  analysis  reporting  of  software 
trouble/problem  reports? 
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11.  SOFTWARE  COST  REPORTING 


a.  Baseline 

1.  Wnat  is  the  baseline  information  from  which  your  software  cost 
reporting  is  developed? 

2.  Provide  this  baseline  description  and  examples  of  its  application. 

b.  Definition  -  Tasks 

1.  Identify  the  tasks  which  are  included  in  the  software  cost 
reporting. 

2.  How  is  the  documentation  effort  costed? 

c.  Software  Types 

1.  Identify  the  types  or  categories  of  software  included  in  the  cost 
reporting,  e.g.,  operational,  development  tools,  support  software,  test  and 
integration  software.  Describe  your  approach  to  assure  that  ALL  software 
which  must  be  developed  is  costed  in  the  proposal. 

d.  Reporting  Vehicle 

1.  Describe  the  process  of  developing  the  software  cost  report 
information. 

2.  Specifically  identify  which  tasks  are  included,  e.g.,  is  software 
QA  included  or  reported  separately? 

3.  What  is  your  criteria  for  reporting  a  cost  or  schedule  variance  in 
software  development? 

e.  Common  Identification  and  Correlation 

1.  How  do  you  control  the  identification  (naming)  of  the  elements  of 
a  total  software  system  to  assure  that  the  software  cost  reporting  is 
traceable  and  can  be  correlated  with  the  technical  products,  schedule  status, 
CWBS,  and  CM  identification  of  the  same  software  system? 

12.  TEST  AND  VERIFICATION 


a.  Plans 

1.  What  is  your  approach  to  planning  for  the  test  and  verification  of 

software? 

2.  Is  this  a  company  standard? 


3.  Wno  writes  the  software  test  plan? 

4.  How  is  this  integrated  with  the  system  test  plan? 
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5.  Wnat  is  your  readiness  criteria  for  start  of  software  test? 

b.  wnat  is  your  baseline  for  start  of  software  test? 

b.  Levels 

1.  How  many  levels  of  testing  is  tne  software  subjected  to  from 
module  verification  tnrougn  system  test? 

2.  Are  tnese  test  levels  formalized? 

j.  wnicn  levels  require  test  procedures? 

c.  Requirements  and  Procedures 

1.  Wnat  is  your  method  to  determine  tnat  tne  software  requirements 
are  testable? 

2.  Wnat  is  your  approach  to  defining  software  test  requirements  are 
testaole? 

3.  Who  writes  and  approves  the  test  requirements  and  procedures? 

4.  wnen  are  tnese  requirements  and  procedures  prepared? 

5.  How  are  tne  test  procedures  verified  prior  to  starting  formal 

tests? 

d.  Tools  and  Facilities 

1.  Describe  tne  tools  and  facilities  required  to  support  the  levels 
of  software  testing  defined  in  "b"  above. 

2.  Do  tnese  tools  and  facilities  exist  to  support  the  subject  program? 
13.  SUFTWAK E  CONFIGURATION  MANAGEMENT* 

a.  Organizational  Approach 

1.  Who  is  responsible  for  software  Configuration  Management  (CM)? 

2.  How  is  software  CM  integrated  with  engineering  and  program 
management? 

b.  baselines 

1.  What  configuration  baselines  are  identified  for  software? 

2.  When  are  these  baselines  established? 

3.  How  are  tne  baselines  controlled? 

4.  How  are  the  software  allocations  structured,  functional  and 
product  baselines  established,  documented,  and  controlled? 
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c.  CM  Identification,  Accounting  and  Control 

1.  How  is  software  identified,  and  controlled  for  accountability? 

2.  Is  the  software  identification  consistent  across  all  disciplines? 

d.  Organizational  Interfaces 

1.  How  does  the  CM  function  interface  with  other  organizational 
elements  participating  in  the  software  development? 

14.  INTERNAL  DEVELOPMENT  STANDARDS 

a.  Management/Development  Status  Control 

1.  Provide  your  standards  for  software  management/development  status 

control. 

2.  Provide  examples  of  recent  application  of  these  standards. 

b.  Documentation 

1.  Provide  your  standards  for  software  documentation  content. 

2.  Provide  your  standards  for  interface  documentation. 

3.  How  do  you  assure  that  these  standards  are  followed? 

c.  Quality  Assurance 

1.  Describe  your  standards  for  software  quality  assurance  (SQA). 

2.  Provide  examples,  e.g.,  a  recent  SQA  Plan. 

3.  (Talk  to  the  QA  department  and  determine  the  specific  standards 
and  procedures  followed  to  implement  software  QA.) 

d.  Work  Definition  Packages 

1.  Provide  examples  of  your  standards  for  software  work  definition. 

2.  Provide  software  work  definition  packages. 

e.  Sizing  Software  Workload 

1.  Provide  your  standards  for  estimating  software  workload. 

2.  Provide  examples  of  estimates. 

f.  Test  and  Verification 

1.  Provide  your  standards  for  software  test  and  verification  plans, 
requirements,  and  procedures. 

2.  Provide  examples  of  test  plans,  procedures  and  reports. 
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g.  Application  and  Enforcement 


i.  How  consistently  are  these  standards  applied? 

Wno  verifies  that  the  standards  are  followed? 

3.  Wno  nas  authority  to  grant  a  waiver? 

15.  SOFTWARE  CONTRACT  WURK  BREAKDOWN  STRUCTURE  (CWBS) 

a.  CWBS  Approacn 

1.  How  is  software  structured  in  the  CWBS? 

2.  How  is  it  correlated  with  the  technical  and  status  reporting? 

3.  Provide  your  internal  standards  for  identifying  software  work 
witnin  tne  CWBS. 

b.  Level  of  Software 

1.  At  wnat  level  is  software  structured  in  the  CWBS? 

What  are  tne  factors  and  criteria  for  determining  the  various 
software  levels? 

3.  At  what  level  (of  CWBS)  is  software  reported  in  the  cost 
performance  report? 

4.  Do  you  require  a  detailed  WBS  from  your  subcontractors?  To  what 
level  do  you  require  software  in  his  WBS? 

c.  interfaces 

1.  How  does  the  software  CWBS  integrate  and  Incorporate  all  of  the 
disciplines  involved  in  the  management  and  development  of  software? 

E.  How  do  you  manage  the  subcontractor  relative  to  his  software  CWBS? 

d.  Application 

1.  Is  software  identified  consistently  within  your  CWBS? 

<?.  Provide  examples  of  your  most  recent  CWBSs  which  include  software, 
lb.  SOFTWARE  WORK  DEFINITION 

a.  Definition  and  Vehicle 

1.  tiow  is  the  software  work  defined  at  the  lowest  level? 

P.  What  vehicle  (work  package  description)  is  used  to  define  the 
software  worK? 

3.  is  this  vehicle  a  standard  within  the  company? 
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**•  Is  the  wopk  definition  directly  traceable  to  the  CWBS? 

b.  Management  Use 

1.  How  do  you  manage  the  software  work  definition,  i.e.,  prepare, 
review,  status,  update,  and  accountability? 

2.  How  is  the  SW  work  definition  used  in  status  tracking  of  software 
development? 


3.  Is  the  work  definition  used  to  identify  work  progress?  How  is 
this  accomplished? 

c.  Cost  Performance  Reporting 

1.  How  is  the  software  work  definition  used  in  cost  performance 
reporting  (CPR)? 

2.  Explain  how  CPR  data  such  as  earned  value  is  derived? 

d.  Application  Examples 

1.  Provide  a  set  of  software  work  definition  descriptions  from  a 
recently  completed  or  on-going  program. 

2.  Provide  a  written  description  of  the  rules  and  interpretation  of 
the  software  work  definition. 

e.  Standards 

1.  Is  the  software  work  definition  defined  by  an  internal  standard? 

2.  Provide  a  copy  of  the  standard. 

f.  Interfaces 

1.  How  does  the  work  definition  interface  and  correlate  with  the 
software  organization  tree,  the  CM  system,  the  software  development 
organization  (staff),  and  the  software  management  systems? 

17.  CONTRACT /CONTROL  METHODS 


a.  Are  there  any  contractual  vehicles  terms  and  conditions  which  you  feel 
are  particularly  effective  for,  or  detrimental  to,  successful  software 
development? 

b.  Rights  to  Software 

1.  Do  you  Intend  to  use  any  proprietary  software  on  the  subject 
program?  Is  this  software  complete?  What  language  is  it  written  in?  How  is 
it  documented?  How  is  It  structured?  How  do  you  intend  to  demonstrate  that 
the  existing  software  you  have  is  one  and  the  same  software  which  implements 
the  support  or  performance  requirements  in  the  final  delivered  system? 

2.  What  are  the  restrictions  to  the  Air  Force  use  of  this  software? 
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c.  Subcontract  Vehicles 

1.  wnat  contractual  terms  do  you  plan  to  use  in  acquiring  software 
from  subcontractors? 

2.  How  will  you  assure  necessary  rights  to  the  software? 

3.  Do  you  plan  to  use  incentives  or  other  contractual  vehicles  to 
motivate  and  reward  software  performance? 

d.  Incentives  -  Award  Fees 

1.  Wnat  experience  do  you  nave  witn  incentives  and  award  fees  for 
software  development? 

2.  Provide  actual  incentive  clauses  wnich  you  nave  used  successfully. 

3.  Provide  evidence  of  positive  incentives  contributing  to  successful 
software  development. 

lb.  SOFTWARE  DOCUMENTATION  APPROACH 


a.  Internal  Documentation  Approach 

1.  Describe  your  approach  and  requirements  for  software  documentation. 

2.  Do  you  use  a  milestone  approach? 

3.  Is  your  approach  compatible  with  MIL-STDs-483 ,  490,  and  1521A? 

4.  How  does  your  software  documentation  interface  with  system, 
subsystem,  hardware,  and  firmware  documentation? 

5.  Whicn  organizations  are  involved  in  generating,  reviewing,  and 
delivering  software  documentation  to  the  acquisition  agency? 

b.  Milestone  Review  and  Approvals 

1.  Wnat  role  does  documentation  play  in  your  internal  reviews  to 
ascertain  software  development  status  and  progress? 

2.  Wnat  is  the  review  and  approval  cycle  for  software  development 
documentation? 

c.  Standards 

1.  Are  your  software  documentation  requirements  defined  in  an 
internal  standard? 

2.  Provide  the  standards  which  define  your  software  development 
documentation  requirements. 
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d.  Application  Examples 

1.  Provide  examples  of  your  software,  documentation  including  CROP, 
Part  I  CPCI  development  specifications,  test  plans,  procedures  and  reports. 
Part  II  CPCI  product  specifications,  Interface  Control  Documents,  ano  otner 

key  software  documents. 

e.  Test  Plans,  Requirements,  Procedures,  Reports 

1.  Define  you r  approach,  requirements,  and  standards  for  software 
test  plans,  requirements,  procedures,  and  reports. 

f.  Specifications 

1.  Wnat  is  tne  role  of  your  software  personnel  in  developing  tne 
system,  subsystem,  and  prime  item  development  specifications? 

2.  How  do  you  allocate  functional  requirements  to  a  software 
specif tcation? 

3.  Who  writes  tne  CPCI  Performance  Specifications? 

4.  How  do  you  assure  that  the  Section  3  performance  requirements  are 
verifiable? 

g.  Subcontractor  Documentation 

1.  Wnat  are  your  internal  standards  for  software  documentation 
requirements  on  subcontractors? 

2.  How  do  you  manage  the  subcontractors  software  documentation 
activities? 

3.  What  incentives  do  you  use  to  assure  the  subcontractor  is 
developing  "quality"  software  documentation? 

19.  SOFTWARE  PRODUCT  AND  QUALITY  ASSURANCE 


a.  Internal  Approach 

1.  What  provisions  do  you  apply  to  assure  quality  software  is 

developed? 

2.  Ho  do  you  implement  the  requirements  of  MIL-S-5277UA? 

3.  Do  you  believe  the  program  required  by  MIL-S-S2779A  contributes  to 
your  objectives  in  software  development? 

4.  Provide  examples  and  results  of  your  internal  software  quality 
initiatives. 
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o.  wuaiity  Assurance  Responsibilities 

1.  wnat  are  tne  responsibilities  and  mission  of  the  quality  assurance 
organisation  regarding  software? 

d.  wno  is  responsible  for  assuring  quality  software?  What  is  the 
role  of  the  software  manager  in  assuring  quality  software? 

c.  organizational  Approacn 

1.  How  is  your  software  QA  function  organized? 

d,  Who  does  software  wA  report  to? 

i.  Is  tne  QA  chain  of  command  independent  of  tne  corporate  program 

manager? 

i.  Application  examples 

1.  Provide  examples  of  your  software  quality  assurance  plans. 

d.  Provide  a  copy  of  your  documented  internal  standards  defining 
procedures  to  implement  software  quality  assurance. 

20.  SOFTWARE  SIZE,  MANPOWER,  SCHEDULE  AND  COST  ESTIMATING 

a.  Software  Size  Estimate 

1.  How  do  you  estimate  software  size? 

2.  Do  you  use  any  models  to  estimate  software  size?  Provide  a  basic 
description  of  tne  software  cost  estimating  model  used  to  support  your 
proposal  and  development  process. 

b.  Software  Manpower  Estimate 

1.  Describe  your  approacn  to  estimating  software  development  effort 
based  on  the  size  of  the  software  to  be  developed. 

2.  Is  tnis  approach  documented?  Provide  a  copy  of  this  document. 

3.  Identify  the  tasks  included  in  your  estimate  and  those  not 
included . 

4.  Do  you  use  a  standard  productivity  factor? 

5.  Is  your  estimation  technique  based  on  a  published  model? 

b.  Wnat  is  your  level  of  confidence  and  basis  for  confidence  in  your 
estima  te? 

7.  Describe  your  model  or  estimating  method  to  decide  how  to 
distribute  tne  total  software  development  manpower  overtime. 
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c.  Schedule  Estimate 

1.  How  do  you  estimate  and  establisn  your  software  development 
schedule  and  milestones  along  the  schedule? 

2.  Is  tnis  scnedule  estimate  based  on  a  published  model'.' 

3-  What  is  your  level  of  confidence  and  basis  for  confidence  ir;  your 
estimate? 

d.  software  Types 

1.  Identity  your  estimation  baselines  for  each  najor  category  of 
software  to  be  developed,  e.g.,  operational,  support,  development  tools  test 
and  simulation. 

e.  Correlation  Among  Size,  Workload,  Schedule  ano  Cost 

2.  How  do  you  correlate  software  estimtes  of  size  (units,  e.g.,  lines 
of  code)  worgload  (manpower),  schedule  (duration,  with  milestones),  and  cost? 

2.  Is  this  correlation  traceable  in  your  management  planning  as 
defined  in  tne  CPDP? 

3.  Who  reviews  the  estimates  and  plan  for  realism  and  consistency? 

21.  INDEPENDENT  VERIFICATION  AND  VALIDATION  (IV&V)  INTERFACES 

1.  Do  you  implement  internal  "independent"  verification  and  validation  on 
your  own  software  development  activities  and  products  and/or  on  your 
subcontractors  software  development? 

2.  Have  you  interfaced  with  any  procurement  agency  IV&V  contractor  on 
previous  programs? 

3.  based  on  tnat  experience,  do  you  think  IV&V  contributed  to  a  higher 
quality  software  product? 

4.  How  would  you  interface  with  a  software  IV&V  contractor? 

5.  Identify  the  documentation  you  will  have  available  to  support  an  IV&V 

effort. 

6.  What  special  prime  contract  contractual  clauses  are  required  or 
expected  to  support  an  IV&V  contract? 
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Section  11.  Language  Waivers 


1.  neview  AFR  800-19,  Vol  I,  AFSC  Sup  1 
AFSC/AFLCR  800-9b  ( ATLAS) 


2.  Review  Figure  II-ll-l  on  guidance  policies 

3.  waiver  is  Figure  II-I1-2. 

9.  Instructions  are  Figure  II-11-3- 
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Figure  II-ll-l  Waiver  Guidance 

1.  Tne  need  for  a  waiver  should  be  identified  during  the  source  selection 
process,  not  after  contract  award. 

2.  If  the  need  for  a  waiver  is  identified,  the  local  language  focal  point 
should  be  contacted  for  assistance.  For  Jovial,  Fortran,  Cobol,  and  Ada, 
ESD/ALEE  is  the  language  focal  point.  For  ATLAS,  ALEQ  is  the  focal  point. 

3.  Language  waivers  are  granted  only  when  compelling  justification  can  be 
Shown.  Justification  must  be  based  on  technical,  system  life-cycle  schedule, 
cost,  and/or  risk  factors. 

U.  The  following  factors  must  be  examined  in  writing  tne  waiver: 

a.  Estimate  direct  development,  test,  and  validation  costs  of  compiler 
and/or  support  tools  if  the  waiver  is  not  granted  for  your  applies*,  io 

b.  Provide  compiler  and  support  tool  delivery  schedules  and  an  evaluatio 
of  the  impact  on  system  schedules  if  your  waiver  is  not  granted. 

c.  Discuss  commonality  of  your  planned  language  with  languages  used  to 
implement  existing  and  planned  system  software  within  the  system. 

d.  Discuss  commonality  of  your  planned  language  with  existing  and  new 
support  software  on  other  systems  that  interface  with  the  system. 

e.  Describe  support  concepts  and  relative  support  costs  over  the  system 
life  cycle  for  each  of  the  languages  being  compared. 

f.  Identify  technical  deficiencies  of  the  languages  being  compared  that 
affect  the  language  decision. 

g.  Identify  changes  required  to  the  existing  AF  approved  HOL  definitions 
to  meet  system  requirements. 

h  Estimate  direct  and  indirect  contractor  costs  associated  with  each 

possible  language  application  (.for  example,  training  or  subcontractor 
cost ) . 

i.  Identify  programming  languages  and  support  tools  used  to  develop  and 
maintain  support  software. 

J.  Discuss  relative  risk  assessment,  with  respect  to  the  system  schedule 
of  using  a  newly  developed  compiler  compared  to  using  a 
vendor-supplied  seasoned  compiler  or  language. 
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Figure  11-11-2.  HUH  OKDEH  LANGUAGE  WAIVER  REQUEST 

Date 


PROJECT  TITLE: _ _ 

on SANITATION  PROCURING  THE  SYSTEM: _ 

UnGAN IT AtiuN  WrtiCH  WIuL  OPtHATE  AND  MAINTAIN  THE  SYSTEM: 


i.  description  uf  the  proposed  hardware  configuration: 

a.  Host/Target/Control  computer,  type,  model,  etc. 


b.  Memory  type: _ Memory  size: 

c.  On-line  System  Peripherals: _ 


d.  Known  available  language  processors  from  manufacturer  or  other  sources 


(e.g.,  FORTRAN,  COBOL,  etc.): 

1.  Assembler 

2 .  Fortran  77  (ANSI  3.9  1978)?  6. 

3.  Cobol  (ANSI  3-2 3)? _  7. 

A.  Jovial  J73? _  8. 

5.  716  ATLAS? _  9. 


e.  Attach  an  ATS  block  diagram  and  a  brief  engineering  description  of  the 
ATS  application  on  a  3eperate  sheet.  (Applicable  for  ATLAS) 


i 
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2.  PROPOSED  LANGUAGE: _ 

a.  Is  there  a  standard  for  this  language?  (ANSI,  IEEE,  DOD) _ 

b.  Is  there  an  existing  compiler/ interpreter  for  this  language 

1.  When  developed? _ 

2.  For  which  system(s)? _ 

3.  Who  maintains  it? _ 

4.  Identify  required  modifications  (if  necessary) _ 

c.  Identify  support  software  modifications  (if  required) _ 


d.  List  of  other  known  USAF  Systems  which  use  the  proposed  language: 


1. 

2. 

3. 

4. 


e.  List  other  USAF  Systems  that  interface  with  this  weapon  system  of  the 
3ame  and  other  maintenance  levels,  and  the  languages  used  for  their  software. 


SYSTEM 


LANGUAGE 


1. 

2. 

3. 

4. 
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3-  TECHNICAL  EVALUATION; 

a.  List  tne  appropriate  approved  HOL  language  problems  and  snow  how  such 
problems  are  solved  with  the  proposed  language.  Use  actual  program  examples, 
(use  separate  page  if  necessary.) 


b.  Can  modifications  to  the  approved  HOL,  use  of  external  procedures,  or 
the  extensibility  of  the  language  result  in  the  same  solution?  Explain  why 
not. 
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LIES  CYCLE  COST  COMPARISON:  (use  15  years  as  tne  system  life).  Attach 
supporting  documentation.  (Each  proposed  HuL  must  be  compared  on  a  seperate 
sheet. ) 


r 


ITEM  COSTS  USING  j 

COMMENTo 

PROPOSED  HOL 

APPROVED  HUL 

COMPILER 

DEVELOPMENT 

MODIFICATION 

MAINTENANCE 

i 

SUPPORT  SOFTWARE 

DEVELOPMENT 

MODIFICATION 

MAINTENANCE 

i 

' 

! 

l 

. 

•SYSTEM  SOFTWARE 

DEVELOPMENT 

MAINTENANCE 

! 

TRAINING 

TOTAL 

(•Software  to  be  developed  using  this  language,  includes  test  programs.) 
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Vo.ume  ii,  nevision  _ 

Contract  Management. 

•  .  oCntutinc  IMHAuf:  Attach  present  program  schedule. 

i'resent  Pnase  of  tne  Acquisition  Cycle _ 

lit'  cooing  nas  already  started,  state  portion  of  program  completed, 
iuomit  samples  of  cooed  modules  demonstrating  tne  advantages  of  the  chosen 
noi..  ) 


l-  l  a  cuss  and  justify  why  the  use  of  an 
programming  mission  critical  software/TPS 


appropriate 
will  impact 


approved  HOI  for 
the  present  schedule. 
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oiuNrT-  l.'A 


.  ion 


TELEPHONE  (commercial ) _ A J  i  V 


for  additional  information  regarding  tnia  waive-  request , 
points  of  contact  are: 


FOE 


NAME 


JhFH’i-  '.Wit’.v 


Unit  under  Test  Ull'l 
< ATuAo 

A i’l:  SYSi&M  (ATLAS! 

?  repose-  Language 
f  ossluie  Hcs*.  System 


Program  -on: rol 
v  oepedti  -  ‘  -  ost 1 

lecrm.  .a.  .Reasons 
Ana  ya  e 

...ife  v.v  •  e  1  ost 
i .imparl.,  -  s 
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8.  TECHNICAL  CONTROL  POINT  COMMENTS : 


NAME: _ 

OHGANUATION: _ 

TELEPHONE  :  (commercial) _ AUTQVON 
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figure  il-iI-3.  Instructions  on  Filling  Out  A  High  Order  Language  Waiver 


Project  Title:  Leif  explanatory 

urbanisation  Procuring  tne  System:  (tSD/Buying  SPO  name) 
organisation  Wnich  Will  operate  tne  System:  (Operating  Command  (POC)) 
organisation  union  will  Monitor  tne  System:  (Tne  designated  support  ALC) 

1.  description  of  tne  Proposed  hardware  Configuration: 

a.  nost.  Target /Control  Computer,  type,  model,  etc.  (Manufacturer,  Model 

i»u' ) 

t.  Memory  type:  (Dynamic  hAM,  BUM,  PKuM,  Disc) 

Memory  site:  (Dumoer  of  Bytes,  words,  (96Kb)) 
r ,  or.  line  system  peripnerals:  (Give  specific  type  of  device,  e.g. 
triennial  pr  inter,  1000LPM  Printer,  6V4  floppydisc,  10MB 
wiricnester  nara  Disc,  etc) 

d.  p.nown  available  language  processors  from  manufacturer  of  other 
sources  (e.g.,  Port  ran,  COBOL,  etc) 

i Languages  supported  by  tne  manufacturer  or  other  sources  for 
for  tnis  hardware  configuration.) 

e.  attach  on  A To  block  diagram  and  a  brief  engineering  description 
of  tne  ATS  application  on  a  seperate  sheet  (ATLAS  Waiver  only). 

(Tne  engineering  description  snould  include  all  physical 
enaracteristics,  how  the  unit  works  basically,  whether  it  is 
programmable  or  manual,  what  software  is  used,  how  it  is  loaded,  how 
tne  unit  interfaces  with  the  UUT,  and  how  tne  unit  interfaces  with 
operator. 

(The  ATS  block  diagram  snould  graphically  represent  the  functional 
hardware  configuration  and  include  signal  interfaces  between  the 
modules/components  (buses).) 

S.  Proposed  Language  Software:  (Language  Name) 

a.  Is  there  a  standard  for  this  language?  (ANSI,  IEEE,  DOD)  (List  tne 
standard  available.) 

0.  Is  tnere  an  existing  compi ler/interpreter  for  tnis  language? 

(Give  which  one  and  manufacturer) 

1.  When  developed:  (Give  original  version/date  and  version  to  be 
used/date. ) 

/.  For  which  systems:  (Give  hardware  configuration  above  version 
presently  runs  on.) 

j.  Who  maintains  it:  (Give  name  of  company.) 

A.  Identify  required  modifications  (if  necessary):  (List  any  and  all 
necessary  modifications  required  to  use  tnis  compi ler/interpreter 
on  tne  proposed  hardware  conf iguration.  Continue  on  a  seperate 
sheet  if  necessary.) 
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c.  Identify  support  software  modifications  (if  required):  (List  any /all 
necessary  modifications  to  the  support  software  in  order  to  use  it  on 
the  proposed  hardware  configuration.  Continue  on  a  seperate  sheet  if 

necessary. ) 

d.  List  of  other  known  USAF  Systems/ATS  which  use  the  proposed  language: 

1.  (List  any  other  fielded  systems  or  automated  test  sets  currently 
in  the  USAF  inventory  which  uses  the  proposed  language.) 

2. 

3- 

4. 

e.  List  other  USAF  that  interface  with  this  weapon  system  of  the  same 
and.  other  maintenance  levels,  and  the  languages  used  for  their 
software. 

System  Language 

1.  (Self  Explanatory) 

2. 

3. 

4. 

3.  Technical  Evaluation: 

a.  List  Approved  HOL  problems  and  show  how  such  problems  are  solved 
with  the  proposed  language.  Use  actual  program  examples  (use  seperate 
pages  if  necessary  ) . 

(Information  here  should  be  explicit.  Problem  definitions  should  be 
clear  and  concise.  Examples  of  code  using  the  proposed  language 
should  be  given  and  explained.) 

b.  Can  modifications  to  an  approved  HOL,  use  of  external  procedures,  or 
tne  extensibility  of  the  language  result  in  the  same  solution.  Explain 

why  not. 

(Answers  here  should  be  objective.  They  should  also  he  more  exp! icit 
than  a  "No."  If  this  area  is  unknown,  state  "unknown",  and  the 
technical  feasibility  concerning  this  area  will  he  determined  nv  the 
ATLAS  technical  control  point.) 

4.  Life  cycle  cost  comparison:  (Use  lb  years  as  the  system  life.)  Attach 
supporting  documentation. 

(Entries  on  tnis  page  must  be  explained  fully.  Explanation  may  be 
given  in  the  comments  section  or  on  an  attached  page  if  more  room  is 
needed.  If  a  software  cost  model  is  used,  full  disclosure  of  ail 
imput/output  data  is  required.  Preferably,  a  'opy  of  the  mode. ’a 
execution  run  snould  be  attached.  Each  proposed  language  snouid 
compared  with  the  appropriate  approved  HOL  on  a  one-to-one  oasis. 
multiple  languages  a  cost  estimate  for  each  language  must  r<*  submit*  •••d. 
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•  'ontract  Kanaseme!.’ 


■enedule  impart:  Attach  present  program  scnedu le. 
i  resent  Pnase  of  trie  Acquisition  Cycle: 

(if  program  coding  nas  already  started,  state  portion  of  program 
completed.  Submit  samples  or'  coded  modules  demonstrating  toe 
advantages  of  tne  trie  cnoseri  tijl..  J 

Discuss  and  justify  why  the  use  of  ar.  approved  HGL  for  software  will 
impact  tne  present  schedule. 

(uiscuss  tally  the  expected  impact  and  snow  now  using  tne  propose'.  Ill 
alleviates  tne  prooletr. . 

Maintaining  organization  Comments  and  Coordination 

line  ALv  designated  to  maintain  tne  nardware  'software  must 
coordinate  and  sign  nere.j 

Contractor  points  of  contact  for  information  submitted. 
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AIK  FORCE  REGULATIONS 


AFR  57-4 
AFSC  Sup  1 

AFK  65-3 
AFSC  Sup  1 

AFR  o6-I2 

AFR  70-15 
AFSC  Sup  1 


REFERENCES 


Modification  Program  Approval  15  Dec  7? 

Hetrofit  Configuration  Changes  1  Apr  '7- 

Conf iguration  Management  11  ,’ul  74 

Configuration  Management  25  Jul  7v 

Aircraft  and  Missile  Equipment  Accour. tab!  1  i ty  15  Aug 

Source  Selection  Policies  and  Procedures  16  Apr  8j 

Source  Selection  Policies  and  Procedures  iR  r'et-  '77 


AFR  60-15 
AFSC  Sup  1 

AFR  60-45 
A ESC  Sup  1 

AFR  300-10 
AFSC  Sup  1 

AFR  310-1 
AFSC  Sup  1 

AFR  600-2 
AFSC  Sup  1 

AFR  800-4 
AFSC  Sup  1 

AFR  800-8 

AFR  800-14,  Vol  I 
AFSC  Sup  1 

AFR  800-14,  Vol  II 


Test  and  Evaluation  12  Sep  SO 

Test  and  Evaluation  19  reu  S. 

Distribution  Statements  on  Technical  Documents  7*  Mar  71 

Distribution  Statements  on  Technical  Documents  22  May  UG 

Computer  Programming  Languages  15  Dec  ?• 

Computer  Programming  Languages  2  Sep  80 

Management  of  Contractor  Data  30  Our,  6^ 

Management  of  Contractor  Data  11  Mar 

Acquisition  Program  Management  14  Nov 

Program  Management  IS  Oct  74 

Transfer  of  Program  Management  Responsibilities  15  Cur.  2.. 

Transfer  of  Program  Management  Responsibilities  II  May  ~’c> 

Integrated  Logistics  Support  Plan  7  fe',  S  > 

Management  of  Computer  Resources  in  Systems  17  Sep  7‘ 

Management  of  Computer  Resources  1 n  Systems  8  An,  7? 

Acquisition  and  Support  Procedures  for  Sep 

Computer  Resources  in  Systems 


AFR  800-19 


System  or  Equipment  Turnover 


27  May  7' 


AFR  800-21 


Interim  Contractor  Support  for  Systems  and 
Equipment 


2-  Sep  "R 


A ESC  Sup  1 


Interim  Contractor-  Support  for  Systems  and 
Equipment 


Jar,  H" 
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AFSC  Regulations  & 
APSCR/AFLC8  80-17 

AFSCR  310-1 
AFSCR/AFLCR  800-2 

APSCP  800-3 

AFSCP  800-7 

Military  Standards  ( 

AFSC  Design 
Handbook  4-2 

MIL-HDBK-339 

MIL-STD-483 
and  Notice  2 

MIL-STD-490 

M1L-STD-1521A 

M1L-STD-1D79(NAVY) 
M1L-S -S2779A 


Pamphlets 

Air  Force  Engineering  Responsibility  for 
Systems  and  Equipment 

Management  of  Contractor  Data 

Management  Multi-Service  Systems,  Programs, 
and  Projects 

A  Guide  for  Program  Management 
Configuration  Management 
Handbooks 


Electronic  Systems  Test  &  Evaluation 


Evaluation  of  a  Contractor's  Software 
Quality  Assurance  Plan 

Configuration  Management  Practices  for 
Systems,  Equipment,  Munitions,  and 
Computer  Programs 

Specification  Practices 

Tecnnical  Reviews  4  Audits  for  Systems, 
Equipment,  and  Computer  Programs 

Weapon  System  Software  Development 

Software  Quality  Assurance  Program 
Requirements 


18  Jul  77 

11  Mar  7- 
4  Sep  73 

9  Apr  76 
1  Dec  77 

10  Apr  71 

15  Jul  81 

31  Dec  70 

30  Oct  68 
1  Jun  76 

1  Dec  78 
1  Aug  74 
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ESD  DOCUMENTS 


ESD-TR-75-85  ADA016488  An  Air  Force  Guide  for  Monitoring  Sep  1975 

Software  Development  Status 

ESD-TR-76-1 59  ADA027051  An  Air  Force  Guide  to  Software  Jun  1976 

Documentation  Requirements 

ESD-TR-77-16  ADA035924  Statement  of  Work  Preparation  Jan  1977 

ESD-TR-77-22  ADA037115  Life  Cycle  Events  Feb  1977 

ESD-TR-77-130  ADA038234  Software  Acquisition  Management  -  Apr  1977 

Software  Development  and  Maintenance 
Facilities 

ESD-TR-77-254  ADA047308  An  Air  Force  Guide  to  Computer  Program  Aug  1977 

Configuration  Management 

ESD-TO-77-255  ADA047318  Software  Quality  Assurance  Aug  1977 

ESD-TR-77-263  ADA048577  Verification  Aug  1977 

ESD-TR-77-326  ADA053039  Validation  and  Certification  Aug  1977 

ESD-TR-77-32  7  ADA053040  Software  Maintenance  Oct  1977 

ESD-TR-78-1 1 7  ADA052567  Reviews  and  Audits  Nov  1977 

ESD-TR-78-139  ADA055573  An  Air  Force  Guide  to  the  Computer  Mar  1978 

Program  Development  Specification 
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